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Thoughts  from  the  Chief  Scientist 


W  it h  the  third  issue  ol  the  Technical  Journal, 
v » >u  should  see  .1  marked  improvement  in  format. 

It  lias  chanced  I’rom  ;t  newsletter  to  a 
piolessional  journal,  with  more  improvements 
lorlheominc. 

The  purpose  ol  the  journal  is  to  disseminate 
technical  information  concerning  test  and 
evaluation,  and  to  discuss  polio  issues  that  arc 
technically  oriented.  C  onceptually.  OTAE  is  a 
simple  mission;  put  the  system  in  a  realistic 
environment  with  operational  personnel  and 
determine  whether  it  is  effective  and  suitable. 

In  other  words,  see  il  it  can  do  its  mission  in 
the  expected  environment  when  operated  and 
maintained  as  planned.  But  simple  in 
implementation  OTAE  is  not.  We  can  seldom 
replicate  the  mission  scenario,  and  must  deal 
with,  tile  normal  uncertainty  and  non- 
repeatability  ol  lield  testing,  problems  ol 
immaturity,  less- than -representative 
conficurations.  etc.  Certainly.  OTAE  is  a 
complex  problem.  Inn  a  tractable  one. 

I  here  are  several  vv  av  \  to  approai  il  ( )TA  I i.  W  e 
can  use  whatever  we  can  find  as  a 
i  c  pi  esentative  environment,  and  then  pul  the 
system  in  that  enviionment  anil  see  what 
h  pp  'fis.  At  the  other  end  ol  the  spectrum,  we 
van  ovale  a  title  operational  environment,  then 
Use  a  statistically  valid  lull  lactorial  test  dcsiun 
with  all  controlled  and  uncontrolled  variables 
completed',  insti  time  tiled.  W  e  shouldn't  do  the 
lot  met.  and  probablv  cau  l  allotd  the  latter,  but 
in  no  i  use  is  out  b  i.  i  i  .,)■  . a.tf  !  ;mg 
acceptable 


dh  e  secret  ol  elleetive  OTAE  is  to  develop  a 
well  delined  and  eonipreliensive  test  concept,  lat 
in  advance  ol  writing  the  test  plan.  The 
concept  should  contain  not  only  the  issues  and 
objectives,  but  also  the  basic  methodology,  test 
scenarios,  how  (if)  simulation  will  be  used  to 
complement  the  lest  and  required  test  resources. 
Embedded  in  the  lest  concept  should  be  a  test 
design  detailing  the  key  system  performance 
factors  (speed,  altitude,  day  nieht.  etc  ),  sample 
si/e.  number  ol  test  replications,  etc.  Integral 
to  the  design  must  be  a  clear  understanding  ol 
why  these  factors  are  important. 

For  example,  suppose  we  choose  a  sample  si/e 
ol  one— a  perfectly  valid  partial  factorial  design. 

W  e  should  be  able  to  support  that  choice  w  it h 
sound  rationale  which  includes  an  analvsts  ol  its 
impact  on  drawing  valid  lest  conclusions.  The 
rationale  may  well  be  that  each  scenario  can 
(>n)y  be  repealed  once.  As  another  example, 
consider  a  system  for  which  the  user 
performance  criteria  requires  that  target 
locations  be  resolved  to  within  .25  nm.  Only  by 
fully  understanding  instrumentation  errors, 
estimated  variance,  and  planned  analysis 
techniques  will  we  be  able  to  identify  lest  design 
strengths  and  weaknesses  to  yield  well-grounded 
conclusions  on  system  ellectiv  eness. 

After  weighing  such  test  design  factors,  we 
may  be  justified  to  request  more  lest  time 
and  or  resources.  At  the  very  least,  we  will 
know  what  to  expect.  The  topic  of  more 
sliuetuied  testing  is  not  merely  academic:  to 
paraphrase  General  Powell.  AFOTEt'  must 
accomplish  more  detailed,  thorough  lest  planning 
and  il  must  be  done  earlier  in  the  lile  ol  a  lest 
program. 

Future  issues  ol  the  Technical  Journal  will 
address  issues  such  as  test  design,  analysis 
techniques  and  other  related  topics.  II  you  are 
familiar  with  specific  examples  of  good  lest 
design  or  even  test  design  problems,  please  share 
them  with  us.  Sound  lest  dcsiun  is  critical  to 
our  business—  elleetive  OTAE. 
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A  Vector-Normalization  in  TOPSIS  That 

Gives  a  Rank  Ordering  Which  is  Invariant 
Under  a  Special  Class  of  Attributes 


By  Srikantan  S.  Nair,  Aviation  Division, 
Operational  Test  and  Evaluation  Agency, 
Falls  Church,  Va. 

INTRODUCTION 

The  Technique  for  Order  Preference  by 
Similarity  to  Ideal  Solution  (TOPSIS)  is  a  method 
used  in  Multiple  Attribute  Decision  Making 
(MADAM)  (1).  It  rank  orders  a  set  of 
alternatives  based  on  a  set  of  attributes  and 
their  associated  weights.  TOPSIS  is  based  on 
the  principle  that  the  alternative  closest  to  the 
ideal  solution,  while  farthest  from  the  negative 
ideal  solution,  is  the  preferred  one.  The  ideal 
solution  consists  of  the  "best"  attribute  values 
available  irrespective  of  the  alternatives.  In  the 
TOPSIS  method  described  in  (1).  the  decision 
matrix  (or  the  matrix  of  attribute  values)  is 
normalized  to  make  all  attributes  the  same  unit 
length  vector  as  well  as  independent  of  the  units 
measurement. 


BASIC  TOPSIS 

A  summarization  of  the  TOPSIS  method 
described  in  ( I )  follows.  If  A[  is  the  i-th 
alternative  and  Cj  the  j-th  attribute  and  Xjj  the 
measure  (quantitative)  of  the  j-th  attribute  for 
the  i-th  alternative,  then  the  decision  matrix  is 
defined  by  (xe).  i  =  I  to  m  and  j=  1  to  n;  m  is 
the  number  of  alternatives  and  n  is  the  number 
of  attributes.  As  a  first  step  of  TOPSIS.  each 
attribute  is  normalized  to  make  all  attributes  the 
same  unit  length  of  vector  as  well  as 
independent  of  the  units  of  measurement.  Thus 
the  normalized  decision  matrix  is  given  by 


In  the  second  step,  the  measures  of  each 
attribute  in  the  normalized  decision  matrix  are 
multiplied  by  its  weight  to  obtain  a  weighted 
normalized  decision  matrix 


U  T 

>ij>  -<  j,  *ii 


i  =  1  to  m  and  j  =  I  to  n. 


(2) 


where  w:  is  the  weight  of  the  j-th  attribute. 
Computational  routines  will  hold  good  even  if  the 
weights  do  not  sum  to  J  since  when  the  final 
ratio  q;  is  computed  in  equation  (5).  the  required 
normalization  factor  will  be  cancelled  out.  Of 
course,  to  maintain  credibility  of  the  TOPSIS 
weights,  it  should  be  standard  practice  to  have 
the  weights  sum  to  I.  The  weighted  normalized 
numbers  \  jj  are  used  to  find  the  set  of  best  and 
the  worst  attribute  values. 


Attributes  are  classified  into  one  of  two  types: 
benefit  attributes  and  cost  attributes.  If  the 
measure  of  an  attribute  is  such  that  larger  is 
better,  then  it  is  classified  as  a  benefit 
attribute.  If  the  measure  is  such  that  smaller  is 
better,  then  it  is  classified  as  cost  attribute. 


The  best  (e.g..  ideal)  attainable  measure  of  a 
benefit  attribute  Cj  is  vt  =  mjix  va.  and  its 
worst  (e.g..  negative-ideal)  attainable  measure  is 


v;  =  mm  v;;.  On  the  other  hand,  the  best 


If 


attainable  measure  of  a  cost  attribute  Ck  is  v£  = 
min  vik.  and  its  worst  attainable  measure  is  \ j”  = 


max  vjk. 

In  the  third  step,  the  separation  measures  of 
each  alternative  from  the  ideal  measure. 


.+  / 


n  .  , 

l  Cvij  -  vT  )2 

j-1  13  3 


i  =  1.2 


.m 


(3) 


fT -2 

*:  •  *  ix  (vu  •  > 


(4) 


i  =  1.2 . m 
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arc  computed.  Finally,  we  calculate  the  relative 
closeness  of  alternative  Aj  from  the  ideal 


•  h  1  (si  +  si> 

i  =  1.2 . m  (; 


where  qj  takes  a  value  between  0  and  i.  Also. 
c|j  equals  1  when  alternative  Aj  has  the  best 
measure  of  every  attribute  and  0  when  Aj  has  all 
the  worst  measures.  Of  course,  it  is  a  rare 
event  to  have  qj  =  1  or  t). 

PROBLEM 


then  \  j  =  maxTjj  is  its  best  attainable  measure 
andTp  =  mill's^  is  its  worst  attainable  measure. 
On  the  other  hand,  if  the  j-th  attribute  is  cost, 
then  vj  =  minTjj  is  its  ideal  andTj^  =  max'vjj  is 
its  neitativc-ideal  attainable  measure. 

II  ^denotes  the  separation  measure  of  the  i- 
th  alternative  from  the  ideal  measure  and'Sj’that 
from  (he  negative  ideal,  then 


'  T  ' 

l  (".« 


i  =  1.2 . m 


There  is  a  special  class  of  attributes  which 
can  be  converted  to  cost  (or  benefit)  from 
benefit  (or  cost)  by  subtracting  its  measure  from 
a  constant.  For  example,  if  an  attribute  is 
measured  in  percent,  by  subtracting  its  measure 
from  l(M).  the  attribute  will  be  changed  from 
benefit  to  cost  (or  cost  to  benefit).  Similarly, 
the  attributes  measured  as  probabilities  when 
subtracted  from  I.  or  attributes  measured  as 
counts  when  subtracted  from  total  counts  where 
total  counts  are  the  same  for  all  alternatives, 
change  ihe  cost-benefit  status.  TOPSIS  rank 
orders  should  not  depend  on  how  this  special 
class  of  attributes  are  recorded,  as  cost  or 
benefit.  The  recording  of  these  attributes  as 
cost  or  benefit  must  be  a  choice  of  convenience. 
In  this  paper  we  propose  a  vector  normalization 
which  gives  a  TOPSIS  rank  ordering  that  is 
independent  of  whether  the  special  class  of 
attributes  are  recorded  as  cost  or  benefit. 

MODIFIED  TOPSIS  METHODOLOGY 

In  place  of  the  normalization  ( I ).  we  propose 
the  following  normalization 

/  /"£  ~  (6) 
(  *ij  -  )  /  J  (* ij  *  XJ) 


which  gives  a  weighted  normalized  decision 
matrix  (7)  in  place  of  (2). 


V  ■  /ill  -V*  (,i 

i  =  I.  2 . m 

Finally  the  relative  closeness  of  the  i-th 
alternative  from  the  ideal  measure  is  given  by 
q;.  where 

q.  «  sT/(s*  +  s7)  • 

'1111  Mir 


i  =  1.2 . m. 

The  set  of  alternatives  can  now  be  rank  ordered 
using  these  q-values.  The  alternative  which  has 
the  highest *q- value  is  the  most  preferred  and 
the  one  with  lowest  q-value  is  the  least 
preferred.  The  rank  ordering  based  on 
Tp  values  w  ill  be  the  same  regardless  of  how  the 
special  class  of  attributes  are  recorded,  cost  or 
benefit. 

NUMERICAL  EXAMPLE 

The  decision  matrix  of  the  navigation  data  on 
four  aircraft  is  given  below.  The  alternatives 
are  the  four  aircraft  A.  B.  C.  and  D  and  the 
attributes  are  the  six  measures  of  performance 
(MOP). 


fX  ij)  •  (  wj  *  f  *1}  *  ) 


’  hue  w:  is  the  weight  and  Xj  the  mean  of  the 
-th  attribute.  If  the  j-th  attribute  is  benefit 


1 


where  MOP4  is  cost  and  all  others  are  benefit 
attributes.  The  weights  of  the  MOPs  arc  given 
by  the  weight  vector: 

w  =  ( .  1676..  1505,.  1657..  1 275. . 2076..  |4o8) 

In  the  above  decision  matrix,  the  attributes 
are  defined  as  follows. 

MOP!:  Percent  courses  completed  (benefit): 
MOP2:  Percent  courses  not  disoriented  (benefit): 
MOP3:  Percent  oi  time  within  2(H)m  of  course 
(benefit):  MOP4:  Sell-location  error  in  meters 
(cost);  MOP5:  Percent  self-locations  within  2(H)m 
( bene  lit):  and  MOP(>:  Velocity  in  knots  (benefit). 
The  attributes  MOPl.  MOP2.  MOP3.  and  MOP5 
belong  to  the  Special  Class  delined  earlier,  since 
each  is  a  percent. 

The  rank  order  obtained  using  the  basic 
TOPSIS  normali/alion  is  A.  B.  C.  and  D  where 
alternative  A  is  most  preferred  and  D  is  least 
preferred.  Their  q-values  were  .71.  .58.  .48.  and 
.40.  respectively.  When  the  decision  maker  (e.g.. 
evaluator)  reclassified  MOP2  and  MOP3  as  cost 
attributes  along  with  MOP4  and  all  others 
remained  benefits,  the  decision  matrix  became 


100. 0  o.o 

90.0  50.0 
100.0  50.0 
100.0  0.0 


29.3  71.1 
J6. 2  <1.3 
59.9  36.0 
45.0  104.5 


The  weight  vector  is  same: 

w  =  ( .  1676,.  1505,.  K)57..  1 275. .21)70..  1468) 

MOP2  is  now  percent  of  courses  disoriented 
(cost)  and  MOP3  is  percent  of 
time  outside  200m  of  course  (cost). 


The  rank  order  now  obtained  in  the  case  (still 
using  the  basic  TOPSIS  normalization)  was  A,  D. 

B.  C.  with  q-values  .80.  .61.  .43.  and  .35. 
respectively.  Thus.  D  moved  to  second  place 
from  the  fourth  place  when  MOP2  and  MOP3 
were  changed  from  benefit  to  cost. 

In  the  case  of  modified  TOPSIS  normalization 
technique  described  in  this  paper,  both  the 
decision  matrices  (11)  and  ( 12)  gave  the  same 
rank  order:  A.  D.  B.  The'q’-values  for  A.  B. 

C.  and  D  in  both  eases  were  .82.  .51.  .47.  and 
.50.  respectively.  It  is  recommended  that  the 
modified  vector  -  normalization  presented  in  this 
paper  be  used  whenever  one  or  more  of  the 
attributes  in  the  TOPSIS  analysis  belong  to  the 
special  class  of  attributes. 

SOFTWARE 

An  in-house  software  package  has  been 
developed  to  compute  the  q  and  q  values  which 
can  be  run  on  an  IBM  PC' or  IBM  compatible  PC 
There  is  no  limitation  on  the  number  of 
alternatives  and  the  number  of  attributes. 

Copies  of  the  software  can  be  obtained  by 
calling  AV  286-2464/%. 
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The  Analytical  Test  and  Evaluation 
Software  System  (ATESS) 


By  Ms.  PaC  L.  Branncn, 

Operations  Analysis  Directorate  and 
Ms.  Kathleen  M.  Hibson, 

ATESS  Program  Manager  BDM  Corp. 

INTRODUCTION 

ATESS  was  developed  for  AFOTEC  to  meet  a 
wide  range  of  typical  T&E  data  reduction  and 
analysis  requirements.  The  need  for  something 
like  ATESS  became  apparent  to  analysts  and 
software  developers  through  experience  gained 
supporting  a  number  of  test  programs,  such  as 
Wild  Weasel  (W'W),  Precision  Location  Strike 
System  (PLSS),  and  High-Speed  Anti-Radiation 
Missile  (HARM). 

Each  test  program  began  with  the  same 
challenge:  develop  data  reduction  and  analysis 
support  software  with  little  or  no  documentation 
ol  the  "real"  data  structures  and  relationships 
that  would  be  available  during  the  lest.  This 
uncertainty  often  delayed  the  start  of  the 
software  development  process  until  it  was  too 
late  to  gel  done  before  the  test  started. 

Even  when  the  software  development  process 
began  early  enough  to  ensure  adequate  lime  and 
resources,  the  lest  program  could  still  be  delayed 
while  software  was  revised  to  accommodate  last 
minute  changes  to  poorly  documented  data 
formats  and  relationships. 

The  same  types  of  software  requirements,  with 
minor  variations,  were  being  identified  and 
addressed  on  different  lest  programs.  Each  new 
program’s  requirements  were  solved  by 
developing  a  new  software  package  tailored  to 
that  single  program.  Many  of  the  processes 
were  common  to  several  test  programs:  but 
specific  data  structure  differences  for  each 
program  required  different  software  solutions, 
preventing  use  of  existing  data  reduction  and 
analysis  software  to  support  new  test  programs 
or  requiring  extensive  revisions  to  meet  new 
constraints. 


CONCEPTUAL  DESIGN 

ATESS  is  a  comprehensive  set  of  modular 
tools,  designed  to  provide  flexible  data  reduction 


and  analysis  support  for  field  tests.  ATESS  can 
be  adapted  to  changing  data  formats  and  flows 
without  changing  the  code  itself.  Rather, 
changes  are  made  through  use  of  external  files 
containing  such  information  as  record  type, 
length,  and  format:  delimiters;  numbers  and  types 
of  parameters:  valid  data  ranges;  functions  to  be 
performed:  and  relationships  between  data  sets 
and  contexts.  In  the  past,  these  types  of 
information  were  usually  embedded  within  typical 
data  processing  and  analysis  software  code, 
requiring  time-consuming  software  revisions  each 
time  one  of  the  items  changed.  The  external 
approach  implemented  in  ATESS  allows  the 
analyst  to  concentrate  on  data  relationships 
rather  than  on  the  mechanics  of  writing  and 
validating  source  code,  relieving  the  analyst  of 
the  overhead  associated  with  developing  software 
to  read,  process,  and  write  data  files  from 
different  sources. 

FUNCTION 

The  indiv  idual  modules  of  ATESS  can  be  used 
in  a  wide  variety  of  combinations  to  produce 
different  processing  flows  to  meet  the  needs  of 
different  test  programs.  At  one  end  of  the 
spectrum.  ATESS  has  a  module  designed  primarily 
to  read  input  data  files  written  on  "other” 
computers  and  produce  output  files  in  "standard'' 
host  computer  files.  At  the  other.  ATESS 
includes  a  module  designed  to  provide  the 
analyst  with  a  flexible  data  display  capability 
(listings  or  graphics).  Other  modules  perform 
such  functions  as  data  quality  control  checks; 
data  file  merging  and  subsetting  under  user 
control:  event  recognition  based  on  lime- 
correlated  multiple  data  file  contexts,  data 
scaling,  and  validation;  manual  file  creation  and 
editing;  and  data  archieval  and  retrieval.  Each 
module  is  independent  of  the  others,  and  there  is 
no  fixed  sequence  for  processing  data  through 
the  individual  modules.  Not  all  modules  need  to 
be  used  for  a  particular  application,  and  each 
module  may  be  used  several  times  in  a  particular 
processing  flow.  ATESS  lends  itself  to  a 
building  block  approach,  initially  implementing  a 
basic  data  flow  to  meet  requirements  at  test 
start,  then  building  to  add  multiple  levels  of 
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data  analysis  as  (Ik-  test  progresses  and  data 
relationships  are  better  understood.  This 

flexibility  is  especially  useful  when  analysis 
emphasis  shifts  during  a  test  from  planned  to 
exploratory,  as  unexpected  relations  come  to 
light,  and  new  questions  must  be  answered. 

STRUCTURE 

AT  ESS  is  a  modular  set  ol  software 
components  that  can  be  used  in  different 
combinations  to  perform  data  reduction  and 
analysis  functions  to  support  TiCE  programs. 
ATESS  consists  of  eight  top-level  functional 
components,  encompassing  approximately  SOD 
routines  or  40.000  lines  of  source  code 
(Figure  I).  Each  functional  component  addresses 
one  data  processing  function  frequently 


performed  in  support  of  T&E.  The  eight 
components  of  ATESS  are  Housekeeping.  Manual 
Forms.  Record  Preparation.  Convert, ''Quality. 
Transformation.  Intervals.  Event  (ieneratoi  and 
Data  Presentation,  as  shown  in  the  overview. 

Each  component  provides  the  user, analyst  with 
several  capabilities,  and  data  may  be  processed 
through  as  many  iterations  of  the  same  or 
different  components  as  required  to  attain  the 
processing  objective  of  the  user.  Several 
capabilities  are  available  in  more  than  one 
component,  prov  iding  even  greater  flexibility  . 

The  order  in  which  components  are  used  for 
processing  d..ta  depends  on  specific  T&E  program 
requirements  and  can  be  tailored  to  meet 
program-specific  data  type,  formal,  and 
processing  requirements  using  process  control 
and  data  description  files. 


Figure  1 

Analytical  Test  and  Evaluation  Software  System 
Overview 
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TOP-LEVEL  COMPONENTS 

Housekeeping  -  converts  ACSII  descriptor  liles 
into  the  required  binary  lorm  lor  use  in  other 
system  components.  Using  the  VAX 
Backup.  Restore  command,  this  component  also 
performs  archieval  retrieval  functions. 

Manual  Forms  -  is  used  to  create  new  data 
tiles  or  to  add.  modify,  or  delete  records  in 
existing  data  lilcs. 

Record  Preparation  -  extracts  and  translates 
raw  data  to  user-specified  parameter  fields.  It 
can  be  used  to  separate  data  into  individual 
records,  retrieve  only  required  data  Irom  input 
disk  or  tape,  and  eliminate  invalid  data  records. 

(  omen  Quality  -  replaces  or  creates  new  data 
\ alucs  In  external  specilication.  It  can  uloballv 
replace  values,  correct  for  calibrations,  convert 
codes,  or  perform  any  user-defined  function 
usinu  external  subroutines,  relational  operators. 
FORTRAN  operators,  and  IF-THEN-ELSE 
statements.  It  also  flat's  questionable  data.  The 
user  defines  the  quality  control  check,  the  flag, 
and  the  replacement  of  the  flag  for  each  check. 
Questionable  data  tire  easily  identified  for 
further  processing. 

Transformation  -  restructures  the  data  files 
and.  or  records  without  changing  the  data  values. 

It  combines  multiple  files  into  one  file,  or  a  file 
can  be  split  into  multiple  files  by  splitting 
records  in  several  ways:  by  outputting  separate 
lilcs  lor  each  unique  value  of  a  selected 
parameter,  by  selecting  subsets  of  the  original 
tile,  by  sorting  records,  or  by  a  combination  of 
sorting  and  subsetting. 

Intervals  -  eliminates  redundant  information 
Irom  lime-generated  (vs.  event-generated)  data. 
The  user  can  define  a  steady  state,  then  specily 
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one  record  for  output  containing  start  and  slop 
times  and  any  support  data  fields  desired. 

Event  Generator  -  reads  and  interprets  the 
contents  of  time-sequenced  multiple  input  files, 
identifying  user-specified  data  states.  It 
performs  the  user-specified  activities  indicated  in 
one  or  more  work  blocks  associated  with  the 
event. 

Data  Presentation  -  produces  listings,  plots, 
and  statistical  summaries.  A  user-entered 
control  file  specifics  the  type,  content,  and 
formal  of  the  output.  This  user  input  has  the 
flexibility  to  produce  various  types  of  listings, 
plots,  and  statistics  using  several  different 
options. 

APPLICATIONS 

The  PLSS  lest  team  was  the  first  user  of 
ATESS  components.  The  software  was  used  to 
reduce,  process,  archive  and  retrieve  data 
gathered  during  limited  IOT&E  of  PLSS.  Data 
Mows  through  ATESS  have  been  defined  for  F-4CI 
\V\\  PL 1 P  which  is  expected  to  begin  testing  in 
FYN.N.  The  remaining  current  users.  Over  The 
Hoi  i/on  Backscatter  Radar  (OTH-B)  East  Coast 
Radar  System  (ECRS)  and  Tacit  Rainbow  (TR). 
have  defined  preliminary  data  flows  and  will  be 
using  ATESS  during  IOT&E  in  the  near  future. 
Additionally.  Consolidated  Space  Operations 
Center  (CSOC).  High-Speed  Anti-Radiation 
Missile  (HARM)  and  MILSTAR  test  teams  as  well 
as  (Tactical  Warfare  Center  (TAWC)  and  Naval 
W  eapons  Center  (NW'C)).  personnel  have  inquired 
into  the  use  of  ATESS.  The  number  of  potential 
users  is  unlimited  because  this  same  software 
could  be  applied  to  virtually  any  new  test 
program,  regardless  of  differences  in  data 
structures  and  methods  of  evaluation.  Thus  the 
use  ol  ATESS  can  significantly  reduce  future 
sol  (ware  dev  elopment  efforts. 
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Coping  with  Mission  Support  Factors 
in  Test  Planning  and  Evaluation 


By  Samuel  G.  Charlton  Ph.D., 
AFOTEC’s  Human  Factors  Division 

INTRODUCTION 

Mission  support  factors  arc  those  elements  ol 
a  system  typically  considered  during  the  course 
of  a  test,  hut  which  are  not  normally  elevated  to 
the  COI  objective  level.  These  mission  support 
(a.k.a.  external)  factors,  contribute  to  mission 
effectiveness  as  necessary  prerequisites  without 
themselves  defining  mission  success.  Mission 
support  factors  include  but  are  not  limited  to: 
in  i  ■;  .abilitv.  compatibility  training,  safety, 
hum.  '  •!  -  oftw  <•■  '  e<  '.!rit'.  and 

surv  iv  abilitv . 

The  importance  of  these  mission  support 
factors  has  become  increasingly  apparent  in  our 
OTiVE  efforts.  A  significant  percentage  of 
service  reports  (NRs)  are  specific  to  the  mission 
support  factors  mentioned  above.  In  addition, 
these  factors  are  receiving  increased  attention 
from  DOTiVE.  Mission  support  factors  are 
important  contributors  to  system  effectiveness 
and  need  h  be  evaluated  ,  ■  such.  The  object  ol 
this  article  is  to  briefly  describe  a  promising 
approach  to  test  planning  and  evaluation  as  we 
have  applied  it  to  the  OTAE  human  factors  ol 
modern  Air  Force  systems. 

FINDING  THE  TARGET 

One  of  the  principal  problems  inherent  in 
evaluating  mission  support  factors  in  modern  Air 
Force  systems  is  the  di  'un  live  and  changing 
characteristics  ol  these  systems.  Owing  to  their 
si/e.  complexity,  and  cost,  many  of  these 
systems,  particularly  in  the  strategic  and 
communications  arena,  are  onc-ol-a-kind. 
evolutionarv  svstems.  Unlike  smaller  products 
where  a  production  decision  can  be  based  on 
extensive  research  and  testing  ol  mission  support 
factors  considerations,  these  large  systems  are 
unique  and  thus  a  decision  to  purchase  one 
comes  first,  followed  In  expenditures  ol  time, 
effort,  and  dollars  to  make  it  work.  The 
development  and  test  cycles  arc  tvpicallv  cut 
quite  short  to  save  money.  As  a  result  there  is 
often  too  little  lime  spent  considering  the 


demands  that  will  be  placed  on  operators  and 
maintainers  by  the  operational  system. 

Adding  to  this  problem  are  the  frequent 
modifications  made  to  the  system  hardware, 
software,  and  procedures.  The  larger  systems 
are  dynamically  reconfigured,  the  commands  and 
procedures  required  to  operate  them  are  changed 
as  their  loads  and  missions  tire  defined  and 
redefined.  The  mission  support  factor 
considerations  of  the  original  system 
configuration  may  bear  little  or  no  resemblance 
to  the  mission  support  factor  issues  pertinent  to 
the  system  alter  having  undergone  extensive 
engineering  and  procedural  changes.  The 
transitional  nature  of  these  systems  also  has  the 
died  ol  making  it  difficult  or  impossible  to 
provide  adequate  training  for  the  operators  and 
maintainers.  there  being  no  static  system  on 
which  to  base  a  training  program. 

THE  PROBLEM  OF  CRITERIA 

A  second,  perhaps  more  fundamental  problem 
inherent  in  evaluating  and/or  assessing  mission 
support  factors  in  OT&E  is  the  lack  of  approved 
standards  or  criteria  on  which  to  base  the 
ev  aluations.  In  some  cases,  technology  advances 
have  outpaced  the  ability  of  industry  to  define 
acceptable  standards  for  software  performance, 
operator  performance,  or  man-machine  interlaces. 
Similarly  .,  many  system  users  are  unable  to 
specify  criteria  lor  these  mission  support  factors 
until  they  have  an  opportunity  to  actually  begin 
to  operate  the  system.  A  related  issue  concerns 
the  selection  of  which  mission  support  factors  to 
test  and  to  what  level  of  detail.  To  this  end.  it 
is  readily  apparent  that  one  cannot  consider 
these  mission  support  factors  in  isolation. 

Mission  support  factors  are  important  only  in  the 
context  ol  their  impact  on  system  effectiveness. 
Further,  the  interactions  between  the  software, 
hardware,  human  operators  and  their  training 
programs  must  be  examined  simultaneously. 

In  the  case  ol  human  factors,  this  issue  arises 
in  the  form  of  lack  of  human  factors  standards 
or  criteria  on  which  to  base  the  OTA  El  ol 
complex  militarv  systems.  While  criteria  do  exist 
lor  the  optimal  phvsical  layout  and 
anthropometry  of  an  operator's  workstation. 
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these  are  not  the  limitin'!  factors  in  operating 
large  C  o  systems.  Tile  pertinent  human  factors 
issues  invoke  identification  of  the  limits  of 
human  operators  in  assimilating  and  responding 
to  the  inloimalion  presented  to  them.  Aside 
from  some  very  basic  knowledge  about  human 
information  processing  and  memory  capabilities, 
lew  criteria  exist  lor  evaluating  the  complex 
inlei  actions  mixing  Irom  mu  it  i  pie  cognitive  tasks. 
Nor  are  there  criteria  lor  determining  the 
correct  way  to  present  the  information  required 
to  support  the  diverse  and  changing  repertoire  of 
operator  tasks. 

AN  INTtCiKA  >  iON  APPROACH 

How  then  are  svstem  evaluators  to  cope  with 
the  aforementioned  problems?  One  promising 
approach  we  are  currently  employing  in  the 
( )TAE  of  several  major  C3  sy  stems  is  based  on  a 
mission  support-system  effectiveness  integration 
approach  to  the  identification  of  mission  support 
factors  risks  in  system  design.  The  integration 
approach  to  human  factors  attempts  to  identify 
i  isk  factors  for  the  performance  E3  operators. 
In  the  absence  of  data  specify  ing  the  causal 
relationships  between  system  design,  human 
information  processing,  and  human  error,  this 
approach  establishes  a  circumstantial  case  for 
the  idcnlil n  ation  ol  situations  and  operator 
tasks  that  are  at  risk  or  closely  associated 
with  system  failure.  This  approach  is  analogous 
to  the  medical  epidemiological  approach 
spccil  ving  risk  factors  associated  with  a  disease 
in  the  absence  of  physiological  data  specifying 
the  cause  of  the  disease. 

Briefly,  the  integration  approach  involves  three 
procedural  steps.  The  first  step  is  the 
identification  ol  which  mission  support  factor 
measures  to  include  in  an  OTiNE  plan.  To 
maximize  the  efficiency  and  effectiveness  of  our 
test  efforts,  mission  support  factor  ()T»VcE 
measures  should  be  selected  on  the  basis  of  their 
relationship  to  system  effectiveness  rather  than 
simply  because  they  are  measurable  or  have 
historical  precedent.  In  essence,  all  our 
measures  should  have  to  pass  the  "so  what?"  test 
prior  to  their  inclusion  in  an  OTiVE  plan.  That 
is.  OTAE  measures  must  cither  possess  uscr- 
specified  criteria,  or  be  logically  and  quantif iablv 
demonstrable  as  having  an  impact  on  the  user  s 
icquiremenls  lor  system  performance. 

for  human  factors  ()TA:E.  we  idcnlil v 


candidate  measures  of  human  performance,  human 
factors  engineering  and  system  performance. 

The  human  performance  measures  include 
operator  response  limes,  decision  times,  and 
response  series  times  and  error  frequencies,  thus 
representing  several  levels  of  task  complexity  . 
These  human  performance  measures  are  drawn 
from  the  human-intensive  components  of  system 
functioning  such  that  they  are  representative  ol 
operator  activities  and  relevant  to  system 
effectiveness. 

The  human  factors  engineering  measures  can 
include  the  results  of  human  factors 
questionnaires  administered  to  system  operators 
or  checklists  completed  by  human  factors 
engineers  evaluating  the  system.  These  human 
factors  engineering  measures  focus  on  issues  of 
display  formats,  data  entry  procedures,  svstem 
documentation,  ambient  noise  and  illumination 
and  other  issues  relevant  to  operator 
performance.  Measures  of  system  performance 
include  system  throughput  times,  failure  rates,  or 
other  measures  related  to  the  user-specified 
requirements  for  system  effectiveness. 

Once  operational  data  of  the  above  three  types 
have  been  collected,  the  next  step  involves 
determining  the  degree  of  association  between 
the  mission  support  factors  and  system 
effectiveness  via  multivariate  statistics.  The  key 
to  this  step  is  linking  the  crileria-less  mission 
support  factors  measures  to  the  objective  criteria 
for  system  performance.  For  example,  when 
system  effectiveness  measures  are  found  to  be 
marginal  or  failing,  measures  of  human 
pcrlormance  are  ranked  according  to  their  degree 
of  relationship  with  those  system  functions. 

Human  tasks  having  a  significant  predictive 
relationship  with  system  performance,  as 
determined  statistically,  are  identified  as  areas 
of  human  factors  focus.  The  human  factors 
engineering  measures  are  then  statistically 
related  to  those  human  performance  tasks  in 
order  to  determine  the  human  factors  issues 
hav  ing  the  greatest  impact  on  the  system 
o pe ra t ors'  per l< tr man ce . 

The  results  of  interest  are  the  human  tasks 
identified  as  having  significant  relationships  to 
system  performance  deficiencies  and  the  human 
I  actors  engineering  issues  possessing  significant 
relationships  with  those  human  tasks.  The 
criteria  or  measure  ol  effectiveness  Tor  human 
factors  engineering  in  these  systems  is  thus 
operationally  defined  as  a  positive  contribution 
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to  overall  system  effectiveness.  C  andidate 
statistical  methods  ramie  from  the  relatively 
simple  multiple  regression  analysis  to  the  more 
sophisticated  factor  analy  tic  and  categorical 
clustering  procedures. 

Finallv.  after  the  relative  degree  ol  association 
of  the  various  human  I  actors  measures  has  been 
determined,  one  can  attempt  to  derive 
explanatory  or  causal  connections  between 
operator  performance  and  the  human  factors 
considerations  ol  the  system  design  and 
operations.  These  explanatory  connections,  can 
then  be  used  to  begin  formulating  more  generic 
sets  of  human  factors  design  criteria  to  be  used 
in  the  design,  test,  and  evaluation  ol  lulure 
s\ stems. 

SUMMARY 

The  integration  approach  to  mission  support 
factors  test  and  evaluation  is  capable  ol 
addressing  many  of  the  problems  identified 
earlier.  The  sy.stcms-lcvel  view  of  the  approach 
ensures  that  our  evaluations  and  assessments  are 
first  and  foremost  directed  at  addressing  user 
requirements  for  system  performance.  All 
mission  support  factors  to  be  assessed  in  ( )TX  E 
are  placed  in  the  larger  landscape  of  their 
impact  on  system  effectiveness.  The  hick  ol 
human  factors  test  criteria  is  also  remedied  by 
using  overall  system  effectiveness  as  the  measure 
ol  [kt  for  malice  by  which  human  factors  issues 
are  judged. 

In  other  words,  areas  of  human  factors  design 


arc  identified  as  being  problems  only  inasmuch  its 
they  are  statistically  associated  with  poor  system 
performance.  Similarly  this  approach  has  the 
advantage  ol  considering  a  variety  of  mission 
support  (actors  simultaneously,  thereby  allow  ing 
the  identification  and  evaluation  of  the 
interactions  between  different  mission  support 
aspects  in  terms  of  their  relative  contribution  lo 
system  effectiveness. 

The  products  of  the  integration  methodology 
also  provide  a  means  for  focusing  remediation 
and  corrective  actions  on  those  areas  w  ith  the 
greatest  impact.  The  specific  mission  support 
factor  problems  associated  with  a  given  system 
function  are  ranked  in  order  of  importance  so  as 
to  provide  a  hierarchy  of  effective  remedies. 
Further,  the  approach  can  provide  periodic 
"snapshots"  of  dynamic  mission  support  factors 
issues  as  they  evolve  in  the  frequently  changing 
env  ironment  of  a  modern  (  3  system. 

Finally,  this  approach  can  serve  as  a  starting 
point  for  the  development  of  a  taxonomy  of 
mission  support  factors  criteria.  Through  the 
systematic  use  of  this  and  similar  evaluation 
techniques,  an  empirical  database  of  mission 
support  factors  standards  can  be  collected  and 
generalized  for  application  to  the  test  and 
evaluation  of  new  systems.  Faced  with  a  need 
to  field  effective  operational  systems,  and  in  the 
absence  of  definitive  human  factors  criteria,  the 
integration  approach  described  above  has  proved 
to  be  quite  valuable  in  the  dynamic  environment 
of  human  factors  lest  and  evaluation. 
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Simple  Linear  Regression  and  Nonparametric 
Slope  Estimation  Techniques  for  Navigation 
Drift  Rate  Caculation  using  Non-Continuous 
Time,  Space,  Position,  information  (TSPI) 


By  Capt.  Robert  A.  Eiscnbcrger  and 
Capt.  Christopher  A.  Strickland 

INTRODUCTION 

The  B-1B  Follow-on  Operational  Test  and 
Evaluation  (FOT&E)  lest  team  evaluates  the 
effectiveness  of  the  B-IB  weapon  system  and  its 
supporting  subsystems.  Two  such  systems  arc 
the  inertial  navigation  system  (INS)  and  the  dead 
reckoning  ( DR)  system.  Although  these  systems 
ate  common  to  many  aircraft,  the  testing  method 
used  by  the  FOT&E  team  is  unique  due  to  the 
instrumentation  systems  restriction  of  non- 
continuous  Time -Space- Posit  ion- In  formation 
(TSPI).  The  purpose  of  this  paper  is  to  describe 
two  techniques  to  estimate  the  true  drift  rate  of 
the  navigation  systems  without  a  continuous  TSPI 
source.  We  will  discuss  two  methods  of 
measuring  the  time  history  of  navigation  position 
error,  two  methods  to  estimate  navigation  drill 
rate  from  the  position  error  history,  and  a 
Monte  Carlo  analysis  which  describes  each 
techniques'  robustness  for  different  quality 
navigators,  and  both  small  and  large  sample 

si/es. 

POSITION  ERROR  MEASUREMENT 

FOT&E  uses  two  sources  ol  measuring  the 
position  error  of  the  navigation  system.  One 
method  uses  Time,  Space.  Position  Information 
(TSPI)  measurements  of  aircraft  position  from  an 
instrumented  range.  These  measurements  are 
assumed  to  be  truth  and  compared  to  aircraft 
recorded  estimates  of  position.  The  difference 
between  these  measurements  is  (he  aircraft 
navigation  system  position  error.  The  second 
method  uses  the  aircraft's  radar  system.  It 
measures  a  range  and  bearing  from  the  aircraft 
to  a  Defense  Mapping  Agency  ( DMA)  surveyed 
ground  point.  This  range  anil  hearing  estimate  is 
added  to  the  known  position  of  (he  surveyed 
ground  point  to  calculate  aircraft  position.  It  is 
then  stored  in  the  aircraft's  computer  complex 


and  compared  to  the  navigation  system  position. 
The  difference  between  the  two  values  is 
automatically  calculated  and  stored  in  temporary 
computer  memory  called  the  "radar  buffer. 

TIME,  SPACE,  POSITION, 
INFORMATION  (TSPI) 

In  discussing  method  one.  a  description  of  the 
aircraft  instrumentation  system  must  first  be 
presented.  The  Production  Data  Acquisition 
system  (PDAS)  is  a  palletized  instrumentation 
system  which  is  installed  in  the  B-IB  Central 
Equipment  Bay.  The  pallet  contains  a  processor 
chassis.  Power  Distribution  Unit,  and  a  disk 
drive  assembly.  The  Disk  Assembly  is  removable 
and  can  hold  up  to  +  100  megabytes  of  data.  The 
PDAS  has  an  internal  clock  which  is  initialed 
with  a  time  synchronization  (WWV)  signal  and 
maintains  time  throughout  flight  within  60 
milliseconds  per  eight  hours.  The  processor 
receives  inputs  from  up  to  six  MIL  standard 
1 55TB  avionics  bus  sources,  reformats  and  edits 
the  data,  and  passes  it  on  to  the  removable  disk 
for  storage.  Following  a  test  mission,  the  disk 
is  removed  from  the  aircraft  and  transported  to 
a  ground  station  where  the  data  is  extracted 
Irom  the  disk  for  further  processing.  L'sing  the 
PDAS.  INS  and  DR  latitude  and  longitude  are 
recorded  continuously  throughout  flight. 

The  PDAS  is  a  passive  listening  device  and 
cannot  act  as  a  TSPI  source.  It  does  not  have 
an  independent  reference  such  as  the  Navigation 
Instrumentation  Accuracy  System  (NAIS)  or  the 
Completely  Integrated  Reference  Instrumentation 
System  (CIRIS).  FOT&E  plans  each  mission  to 
pass  through  the  radar  coverage  of  multiple 
instrumentation  ranges  along  the  route  of  flight. 
Coverage  over  each  range  lasts  from  2  to  30 
minutes  depending  on  mission  profile,  range 
availability  and  range  capabilities.  Due  to  the 
limited  number  of  ranges  available  on  any  given 
day.  we  receive  one  to  four  short  periods  of 
continuous  TSPI  and  long  periods  of  no  TSPI.  A 
position  error  time  history  of  these  portions  of 
continuous  TSPI  coverage  is  calculated  posl- 
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I  light.  The  calculation  uses  an  oblate  (assumes 
the  earth  is  elliptical)  earth  model  to  convert 
differences  in  latitude  and  longitude  to 
differences  in  feel  (3). 

The  errors  associated  with  using  this  technique 
of  position  error  time  history  measurement  are 
as  follows: 

Timing  difference  between  the  PDAS  clock 
reference  and  range  time  reference.  Both 
sources  synchronize  to  WWV.  The  range  uses  an 
interrange  instrumentation  group  (IRI(i)  lime 
generator  to  maintain  the  reference,  while  the 
PDAS  uses  an  internal  crystal.  Since  the  PDAS 
can  drill  up  to  60  milliseconds  after  eight  hours, 
a  position  error  is  created  due  '  '  th'*  aircraft 


position  and  site  alignment  inaccuracies.  Though 
actual  capability  varies  from  site  to  site,  an 
estimate  of  100  feet  of  error  bounds  all  systems. 

Oblate  ellipsoid  model  assumes  the  earth  is  an 
ellipse  w  ith  a  specified  eccentricity.  Though  this 
is  not  totally  accurate,  for  short  distances  (  <  20 
miles)  this  error  is  small.  FOTiSiE  testing  shows 
it  to  be  approximately  It)  feet. 

Therefore,  summing  the  values  from  a.b.  and  e 
above,  the  error  in  measuring  position  error  bv 
comparing  aircraft  recorded  position  to  range 
measured  |X)sition  is  on  the  order  of  plus  or 
minus  j- ISO  feet. 

An  example  of  the  data  accumulated  using  this 
technique  is  plotted  in  Figure  1. 


Figure  1 
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airspeed  during  this  60  millisecond  lapse.  With 
an  average  cruise  speed  of  750  It/second.  typical 
errors  due  to  time  are  approximately  45  feet  (75(1 
It, sec  .06  seconds). 

Range  measurement  error  is  the  ability  of  the 
instrumentation  range  to  measure  aircraft 
position.  This  error  includes  errors  due  to 
tracking  the  aircraft  at  other  than  the  INS 
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RADAR  BUFFER  READINGS 

As  stated  earlier,  the  second  method  of 
accumulating  a  time  history  of  navigation 
position  error  uses  the  aircraft  radar  system.  In 
Figure  2.  the  nav  igator  commanded  a  radar  map 
of  an  area  that  contains  a  surveyed  point.  For 
this  example,  the  point  is  the  center  of  a  bridge 


spanning  a  river  between  two  land  masses.  Each 
poini  on  live  radar  map  has  a  latitude  and 
longitude  associated  w  ith  it  as  measured  In  the 
INS.  By  overlaving  a  sel  ol  moveable  computer 
generated  crosshairs  on  the  map.  the  nav  igator 
has  the  ability  to  determine  his  position  error. 

The  procedure  follows: 

a.  The  map  is  generated  as  in  Figure  2  when 
the  navigator  commands  the 

INS  to  place  the  crosshairs  on  what  it  thinks 
should  be  the  aimpoint. 

b.  lit  the  example,  the  INS  placed  the 
crosshairs  northwest  of  what  the  radar  system 
shows  to  be  the  aimpoint  (center  of  the  bridge). 

c.  Since  the  INS  commanded  (uncorrected) 
placement  of  the  crosshairs  is  in  error,  a 
position  error  exists  in  the  INS. 

d.  The  navigator  moves  the  crosshairs  on  the 
aimpoint  through  a  hackhandle  control  slick  with 
the  result  being  Figure  3. 


c.  On  the  lower  left  corner  of  the  map.  the 
radar  buffers  tire  displayed.  The  lop  number 
represents  the  error  in  the  INS  in  distance, 
while  the  bottom  number  represents  the  bearing. 
These  bullers  tire  accurate  for  the  time  of  map 
generation,  displayed  on  the  lower  right  corner 
of  each  map.  These  numbers  are  only  accurate 
when  the  crosshairs  are  on  the  aimpoint. 

(Figures  2  &  3) 

This  information  is  hand  recorded  on  a  lest 
card  every  15  minutes  by  the  navigator.  After 
assuming  that  the  Schuler  frequency  is  the 
highest  frequency  in  the  signal,  a  sampling  rate 
can  be  determined.  Fifteen  minutes  of  sampling 
is  required  to  satisfy  Shannon's  sampling  theorem 
which  states  "A  band  limited  signal  can  be 
uniquely  represented  by  a  set  of  samples  taken 
at  time  intervals  spaced  less  than  1/2  W  seconds 
apart,  where  W  is  the  highest  frequency  in  the 
signal  |5|."  Since  the  frequency  of  the  Schuler 
period  is  S4.4  minutes.  Shannon  requires  samples 
to  he  taken  at  least  every  42  minutes. 


ALTITUDE  (FT) 


BUFFER  README 
DISTANCE  (FT) 
BEARMG  (DG) 


RADAR  DISPLAY  WITH  UNCORRECTED 
CROSSHAR  PLACatENT 
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Figure  5 


Using  this  technique,  a  position  error  time 
history  plot  can  he  created  that  has  one  sample 
every  15  minutes  throughout  the  mission.  An 
example  is  plotted  in  Figure  4. 

Figure  4 
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DRIFT  RATE  ESTIMATION 


The  errors  associated  with  using  this  technique 
of  position  error  time  history  measurement  are 
as  follows: 


SIMPLE  LINEAR  REGRESSION  (SLR) 

The  most  common  method  of  estimating  a  drift 
rate  is  to  use  the  slope  ^  from  the  SLR  model: 


Avionics  operator  crosshair  placement  error  - 


This  is  the  ability  of  the  navigator  to  identify  ( 7 ) 

an  aimpoinl  on  the  radar  display  unit  and 
position  the  crosshairs  exactly  on  the  lop  of  it. 

It  also  includes  the  error  due  to  picture  |n  ^7^  v 

distortion,  beam  smearing  and  antenna  •. 

misalignment.  This  error  has  been  measured  &o 

through  flight  test  and  is  shown  to  be  <■> 

consistently  less  than  200  feet  |f>|.  3| 


y  ,  0O  *  pjt  *  E, 
vhere  E  -  •  (0  , o' ) . 

the  fitted  data  point. 

the  v  intercept. 

the  slope. 


Target  survey  measurement  error  -  The 
Defense  Mapping  Agency  publishes  measurement 
errors  associated  with  each  aimpoint.  Aimpoints 
with  50  feet  error  or  less  are  used  for  our 
analysis. 


£ 


4(0.  <rM 


the  difference  between  the 
fitted  and  the  actual 
value, 

normal  distribution  with  a 
mean  of  zero  and  some 
unknown  variance  (cr1). 


COMBINING  TECHNIOUES 

Whenever  possible,  a  combination  of  the  two 
position  error  measurement  techniques  are  used 
to  accumulate  a  position  error  lime  history  (see 
Figure  5). 

I _ I - 
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SLR  is  a  method  of  fitting  a  linear  model 
through  a  set  of  points.  The  model  is  lit  by 
the  method  of  least  squares.  In  least  squares 


regression,  (he  mode!  is  (he  line  with  (he 
smallest  squared  deviations  between  points  on 
the  regression  line  and  the  actual  data  points 
(see  Fieure  (>)• 


S  i  ■  P  * «  .ntir  Hrjrtsiun  1.  •  •  f  :  5  j  j  t  r  c  r  :  ' 


THE  THEIL  STATISTIC 

The  next  method  is  a  better  estimator  of  drill 
rates  when  the  sample  sizes  are  small  and  the 
data  is  erratic.  It  is  called  "An  Estimator 
Associated  With  the  The i  1  Statistic"'  by  Theil. 

The  model  built  is  similar  to  the  model  used  in 
equation  (7).  However,  normality  of  the  errors 
is  not  required.  The  only  assumptions  are  as 
follows: 

a.  The  errorsE  are  mutually  independent. 

b.  Each  g comes  from  the  same  continuous 
population. 

This  method  has  practical  test  applications  when 
estimating  drill  rates  since  it  uses  all  possible 
slopes  (see  Figure  7). 


Figure  7 

Slope  Estimated  From  The  Theil  Statistic 


This  method  taki 1  v  -  of  all  of  the  "wjiat  if" 
situations.  For  example  what  if  I  started  at 
point  b  and  stopped  at  point  d.  what  would  mv 
drift  rate  be?  The  Theil  Statistic  for  Figure  7 
is  the  median  of  slopes  {sj.  si.  S3.  S3.  S4.  S3}. 

It  should  be  noted  that  the  estimator  of  slope 
from  the  Theil  statistic  is  less  sensitive  to  gross 
errors  than  is  the  classical  least-squares 
estimator  from  the  SLR  method.  This  is  because 
the  Theil  Statistic  lakes  the  median  of  the 
slopes,  w  hereas  the  classical  slope  is  computed 
by  a  weighted  average  of  the  slopes  |7|. 

MONTE  CARLO  ANALYSIS 

To  determine  the  robustness  of  each  drift  rate 
estimation  technique,  a  Monte  Carlo  analysis 
using  48(H)  samples  was  performed.  This  analysis 
varied  as  lit 

a.  Schuler  amplitude 

b.  Drift  rate 

c.  Position  error  sample  size 

Each  of  these  parameters  were  first  varied 
separately,  i.e.  Schuler  amplitude  was  varied 
while  drift  rate  and  sample  si/e  remained 
constant.  Additionally,  two  variables  were  varied 
while  holding  the  third  constant.  This  approach 
allowed  us  to  determine  the  ability  of  each 
estimation  technique  for  different  types  of 
navigators  and  different  data  collection  systems. 

SCHULER  AMPLITUDE 

The  first  variable  investigated  was  Schuler 
amplitude  (4).  A  model  depicting  this  84  minute 
error  is  shown  in  Figure  8. 

Figure  8 

Schuler  Tuned  Inertial  System  Error 


Pos i t i on 
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else 


The  amplitude  "A"  above  varies  with  different 
quality  navigators  or  aiding  techniques.  The 
lower  the  platform  misalignment  error  (better  the 
INS),  the  lower  the  Schuler  amplitude.  For 
aided  systems  (i.e.  doppler  aided,  stellar) 
misalignment  error  is  periodically  reduced,  thus 
decreasing  the  Schuler  amplitude. 

DRIFT  RATE 

The  model  incorporated  different  drift  rates 
with  these  values  representing  systems  with 
strategic  accuracy,  such  as  an  IC'BM.  through 
coarse  navigation  schemes  used  in  civil  aviation. 

POSITION  ERROR  SAMPLE  SIZE 

This  parameter  was  varied  to  estimate 
realistic  sample  si/e  opportunities.  This  sample 
si/e  was  selected  randomly  from  the  3<>()  position 
points  generated  from  the  six  hour  navigation 
leg.  Though,  one  sample  every  I  s!  minutes  is  the 
minimum  recommended  sample  from  the  total 
population,  we  will  show  smaller  sample  si/es  can 
be  used  to  estimate  an  accurate  drift  rate. 

RANDOM  ERROR  MODEL 

The  simulation  involved  4800  six  hour 
navigation  legs.  Each  time  a  nav  leg  was 
generated,  it  computed  one  sample  every  minute 
lor  a  total  of  300  samples  per  navigation  leg. 

The  nav  legs  were  based  on  a  Shuler  with 
amplitudes  of  250.  500.  750.  1000.  1500  and  2000 
feet  and  drift  rates  of  .1.  .3.  .5.  .75,  I.  1.5.  2 
and  3  nm,  hr.  From  each  nav  leg  random 
samples  ol  5.  It).  15.  25  and  30  were  drawn  from 
the  300  points  in  the  navigation  leg.  Twenty 
nav  leus  were  generated  at  each  amplitude,  drift 
rate  and  random  sample  si/e  grouping.  The 
results  ol  each  of  the  groupings  can  be  seen  in 
Tables  I  -  ’ 

The  lormula  lor  determining  the  model  is  as 
follows: 

(8)  /(  T,«:t)  -  Tt  .  Ssin(.0744t)  .  C(t) 
where 

(9)  c(l)  is  defined  as  follows: 
if  U(0,1)  <  .95  then 

e(t>  -  Ut-300,3 00) 
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If  z  .  150- *(0, 1)  >0  then 
e( t )  -  300  *  z 


else 

c ( t )  ■  -300  +  z 


* 

i  s 

a  normal  d 

istribution 

u 

i  s 

a  uniform 

distribution 

T 

i  s 

the  shuler 

amplitude  in 

feet. 

5 

i  s 

drift  rate 

in  feet  per 

minute 

t 

i  s 

time  in  mi 

nu  t  e s  . 

/ 

i  s 

the  drift 

in  feet. 

E( 

t) 

is  the  mod 

el  error 

RESULTS 


( Iverall  Comparison:  Theil  Statistic  vs  SLR. 

An  analysis  of  variance  (ANOVA)  was  applied  to 
all  of  the  test  conditions  against  the  following 
hypothesis: 

H0  -  There  is  no  difference  between  the 
slope  in  the  Theil  Statistic  and  the  slope  from  a 
simple  linear  regression  (SLR)  for  estimating 
drift  rates. 

FL,  -  The  slope  from  the  Theil  Statistic  is  a 
better  estimator  of  drift  rates. 

Based  on  the  results  of  the  ANOVA.  reject  Fl0 
in  favor  oT  H.(.  Over  all  three  factors 
investigated  an  estimated  drift  rate  based  on  the 
Theil  slope  gives  you  an  error  of  .0032  nm/hr: 
while  the  SLR  slope  gives  you  an  error  of  .0040 
nnVhr. 

By  Sample  Si/e:  Theil  Statistic  vs  SLR. 

From  Table  I  it  can  be  seen  that  in  most  cases 
the  Theil  Statistic  was  a  better  estimator  of 
drill  rate  than  the  SLR  technique,  though  the 
two  estimators  lend  to  converge  as  the  sample 
si/es  increase. 


Tabla  t 

l trot  In  drift  rat*  ••ti**rion  by  Saapl*  Slits 
baaad  an  *60  n*v  lags  par  itapl*  ant 
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1  V"0l 

Bv  Drill  Rato:  Theil  Statistic  vs  SLR. 

Table  2  shows  the  Theil  Statistic  to  he  a  better 
estimator  ol  drill  rate  than  the  SLR  technique. 
The  error  between  the  two  techniques  is 
constant  over  all  drill  rates  indicating  H’e 
accuracy  ol  the  slope  technique:,  is  unaffected  bv 
this  variable. 


Pvalue  is  the  smallest  probability  that  we 
can  re  ject  the  hypothesis  that  the  Theil  slope 
equals  the  regression  slope.  (Typically  we  would 
reject  this  hypothesis  for  anv  Pvalue  <  •05) 


measurement  of  the  error.  Front  these  lime 
histories,  drift  rate  is  calculated  by  using  an 
estimator  of  slope.  Two  techniques  were 
presented  to  derive  this  estimator.  The  first 
method,  simple  linear  regression,  proved  to  be 
accurate  within  .004(>  nm/hr  average  error.  As 
sample  rate  increases,  performance  of  this 
estimator  gets  belter,  however  the  more  the 
Schuler  cycle  is  allowed  to  remain  undamped,  the 
poorer  the  performance.  The  second  method 
used  the  Theil  statistic  to  estimate  drift  rale.  It 
proved  to  be  a  more  robust  estimator  under 
virtually  all  lest  conditions.  Its  average  error  is 
.0052  nm/hr. 


Error  tn  drift  rata  oatmclon  by  Drift  Rato 
Hied  on  600  nv  lift  ?#r  drift  rat* 
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Bv  Shuler  Amplitude:  Theil  Statistic  vs  SLR 
As  in  the  other  two  tables,  the  Theil  Statistic  is 
the  better  estimator  of  drift  rate.  There  are 
noticeable  trends  in  the  difference  between  the 
two  methods  as  the  Shuler  Amplitude  increases. 
With  an  increase  in  amplitude,  both  estimation 
techniques  increase  their  error;  however,  the 
SLR  technique  is  increasing  at  a  higher  rate. 


REFERENCES 

1.  Offensive  Avionics  Production  Data 
Acquisition  System  (PDAS)  Users  Manual. 
Rockwell  International.  Document  #  NA-80-1388. 
17  October  198<>. 

2.  Production  Dalit  Acquisition  System  (PDAS) 
users  Manual.  Boeing  FSCM  No.  82918.  Document 
#  D4(  10-4500 1  -7.  23  April  198<>. 

3.  U.S.  Coast  and  Geodetic  Survey  Special 
Publication  Number  241. 

4.  American  Practical  Navigator.  Pub.  No.  9, 
Bowditch.  1977. 


I  r»  drift  r 
t*  *90  in 


it  ten  V/  a  *  p  .  |  '  i  4  « 
S  C  1  i  i  «  ?  4  *  0  :  ;  '  . 4 ' 


i i t 

-  i*  Thfi;  Errsrl 

"•in  ! 

SLR  Error 

lean 

■  :n«  . .  -  j;.* 

"em 

9003 

7919 

i  -  7  5 

}n0 

f  7014  ) 

70  2  4 

i  -  7919 

"  *  9 

702  4  j 

70  38 

1  -  ) : :  - 

:  •'■'M 

i  70  2  8 

09.  J 

-  7013 

:  *  '0 

;  0952  1 

90 ’9 

-  c :  8 

‘  M 

:  :oo 

7065  | 

.  9942 

-  :22’ 

'99  1 

5.  Continuous  and  Discrete  Signal  and  System 
Analysis.  Me( iillem  and  Cooper.  Holt.  Rinehart 
and  Winston.  Inc..  1974. 

6.  Monthly  Operational  Suitability  and 
Effectiveness  Summary.  B-1B  Follow-on 
Operational  Test  and  Evaluation  Test  Team,  30 
November  1987. 


CONCLUSIONS 

The  B-1B  Follow  On  Test  and  Evaluation  Test 
Team  must  perform  navigation  effectiveness 
testing  without  a  continuous  TSPI  source.  Two 
methods  were  presented  to  measure  the 
navigation  error  lime  history  of  the  INS.  One 
method  uses  an  on-board  recorder  to  record  INS 
position  and  compare  it  to  noil-continuous  TSPI 
sources  to  calculate  the  error.  The  second 
method  uses  the  aircraft  radar  system  as  a  direct 
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Tactical  Operational  Test  and  Evaluation 


By  Mr.  Donald  L.  (iiadrosich, 
USAFTAWC  Chief  Scientist 

INTRODUCTION 

Operational  testing  can  he  defined  in  main 
wavs.  hut  central  to  all  of  the  definitions  is  the 
acquisition  ol  information  to  support  acceptance 
or  rejection  ol  an  idea,  concept,  or  weapons 
svstem.  ( fperational  testing  establishes  or 
reinforces  the  confidence  of  decision  makers  that 
a  system  is  indeed  ready  for  production  or 
application  in  combat.  For  those  systems  that 
are  judged  to  he  ready,  it  refines  their 
performance  by  identifying  what  can  be  done  to 
make  them  even  better,  and  for  those  that  aren't 
ready,  it  should  explain  why  and  whether  they 
can  eventually  be  made  ready.  For  most  complex 
tactical  systems,  acceptance  or  rejection  is 
rarely  a  clear  cut  issue  totally  supportable  by 
quantitative  data,  and  the  informed  judgment  of 
the  operator  must  play  a  vital  role  in  the 
decision  process.  This  article  focuses  upon  some 


to  the  difficult  tradeoffs  made  in  operational 
testing  and  the  important  roles  played  by  the 
operational  users,  testers,  and  developers. 

SYSTEMATIC  APPROACH 

For  over  20  years  tactical  operational  testers 
have  used  a  systematic  approach  in  the  design 
and  execution  of  operational  tests  and  tactics 
development  projects.  Figure  I  depicts  a 
simplified  version  of  this  approach.  We  start  with 
a  purpose  that  is  amplified  by  the  objectives. 

The  method  of  test  is  selected  based  on  how  the 
idea,  concept,  or  system  relates  to  the 
appropriate  tactical  air  missions  and  the  expected 
threat  environments.  For  each  objective,  one  or 
more  measures  of  effectiveness  are  chosen  to 
assess  performance  with  regard  to  that  objective. 
Both  qualitative  and  quantitative  measures  are 
important  and  are  used  with  emphasis  upon 
operational  realism,  relevance,  sufficiency, 
accuracy,  and  achievability. 


FIGURE  1 

SYSTEMATIC  APPROACH  TO  TACTICAL 
TESTING/TACTICS  DEVELOPMENT 


The  measures  of  effectiveness  lead  lo  ihe  data 
that  must  he  collected  during  the  lest  and  to  the 
instrumentation  and  methods  that  need  lo  he 
used  in  collecting,  processing,  and  summarizing 
the  data  lor  analysts.  When  decisions  are  made 
in  a  test  report,  the  test  report  should  define 
the  criteria  that  explains  the  basis  of  the 
decisions.  (  'rilcria.  in  turn,  should  he  based 
directly  upon  the  operational  need  which 
considers  both  friendly  and  enemy  capabilities. 
Since  most  systems  require  a  long  time  in 
development  and  capabilities  change  with  time, 
the  criteria  must  reflect  the  latest  inlorniation 
lo  ensure  (hat  the  system  produced  can  meet  the 
threat.  The  most  appropriate  time  and  place  for 
linal  commitment  lo  criteria  is  after  the  decision 
maker  has  had  the  benefit  of  the  knowledge  and 
experience  gained  during  operational  testing. 

TEST  DESIGN  ISSUES 

In  today's  complex  testing  world,  the 
operational  tester  always  faces  the  problem  of 
trying  lo  decide  what  to  physically  lest  and 
what  to  simulate.  On  one  hand,  there  is  the 
urge  to  he  operationally  realistic  by  dropping 
bombs,  shooting  bullets,  and  so  lorlh.  On  the 
other  hand,  computer  simulations  can  examine 
many  situations  that  can't  be  physically  tested, 
and  often  simulation  can  be  done  at  significantly 
less  cost.  In  some  cases,  a  combination  of 
physical  testing  and  computer  simulation  is 
required  to  examine  the  lull  spectrum  of  issues, 
threats,  etc. 

In  my  view,  there  is  no  substitute  for  physical 
operational  testing  if  it  is  within  the  realm  of 
being  possible  and  practical.  In  those  cases 
where  it  is  not  possible  for  physical  operational 
tests  to  satisfy  the  issues,  computer  simulations 
and  other  analytical  practices  may  be  in  order. 
However,  the  bottom  line  is  that  if  you  can  fly 
it  and  test  it.  that's  the  most  desirable  and 
credible  process.  The  operator  needs  the 
assurance  provided  by  physical  testing  that  (he 
actual  weapon  system  ti  e.,  not  the  computer 

simulation)  will  work  given  that  he  has  to 
employ  it  in  combat. 

In  any  test  design,  there  is  always  the 
question  on  how  much  data  tire  enough.  One 
answer  lo  this  question  is  probabilistic,  in  that 
it  has  to  do  with  how  much  risk  the  decision 
maker  is  willing  to  accept  regarding  the  "proof" 
or  "disproof”  of  the  idea,  concept,  or  svstem.  A 
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confidence  level  of  at  least  .HO'/  should  be 
demanded  by  any  prudent  operator  where 
quantitative  measures  are  applicable  and  the  risk 
can  be  quantified.  Regardless  ol  whether 
quantitative  or  qualitative  measures  are  being 
used,  the  sample  si/e  must  be  large  enough  to 
conv  ince  the  operational  expert  that  the  results 
are  valid.  If  this  is  not  possible,  then  the  tester 
needs  lo  find  more  credible  ways  and  measures 
to  make  the  evaluation. 

Operational  testers  should  be  highly  sensitive 
to  the  need  for  the  right  balance  ol 
instrumentation  versus  no  instrumentation.  II 
the  operational  tester  can  understand  what's 
going  on  by  rely  ing  upon  operator  observations 
augmented  by  a  minimal  amount  ol 
instrumentation,  that's  the  way  it  should  be 
done.  Any  time  instrumentation  is  introduced 
into  the  measurement  process,  there  is  also  an 
increase  in  the  cost  and  time  involved  in  the 
test,  a  corresponding  risk  that  the  measurement 
system  may  adversely  affect  the  environment, 
and  a  tendency  of  the  tester  to  rely  upon  the 
numbers  generated  rather  than  tactical 
operational  judgment.  Too  little  instrumentation 
can  result  in  not  understanding  whal  s  going  on 
in  the  lest:  whereas  too  much  instrumentation 
mav  bias  the  results  and/or  make  it  impossible  to 
complete  the  test  at  a  reasonable  cost  and 
within  a  reasonable  time  frame. 

OPERATIONAL  TEST  INDEPENDENCE 

A  lot  has  been  said  in  recent  years  about  the 
need  for  more  independence  in  operational 
testing.  This  immediately  should  lead  one  lo  the 
question:  independence  from  what?  On  a  new 
system,  the  developer  beyond  question  has  a 
vested  interest  in  'making  the  system  work"  and 
he  also  has  the  most  knowledge  about  "how  the 
svstem  works."  As  taxpayers  and  interested 
citizens,  most  of  us  are  reluctant  to  see  a 
military  system  scrapped  tiller  several  years  and 
millions  of  dollars  have  been  expended  in  its 
development.  On  the  other  hand,  whether 
developer,  tester,  or  user,  we  must  demand  and 
get  a  system  that  satisfies  the  current  needs  ol 
the  user.  Anything  less  is  shirking  our 
responsibility  and  is  not  acceptable. 

There  also  litis  been  a  rather  loose  use  ol  the 
term  "operational  user."  The  operational  users 
clearly  are  the  operational  commands  which  will 
eventually  receive  the  systems  and  lake  them 


into  combat  il  the  need  arises.  There  mav  be  a 
reason  tor  some  independence  by  the  tester  from 
the  developer  to  ensure  objectivity  in  the 
evaluation;  however,  arguments  regarding  leslei 
independence  Irom  the  operational  user  is  like 
Irving  to  select  the  automobile  you  like  for  vour 
neiehboi 

SUMMARY 

Acquisition  and  initial  testing  of  weapons 
svstems  today  is  more  ol  a  team  process  than  it 
ever  has  been  in  the  history  of  the  Air  Force. 

The  savvy  ol  the  developer,  the  independence  of 
the  lest  command,  and  the  operational  expertise 
ol  the  user  are  all  essential.  The  informed 
judgment  ol  the  operator  based  on  physical  test 
data  and  experience  has  always  been  and  still  is 


the  most  powerful  ingredient  of  operational 
testing.  Computer  simulations  and  other  forms 
of  analysis  are  used  where  appropriate  to 
augment  and  supplement  the  results  of  physical 
tests. 

The  operational  commands  are  the  primary 
organizations  responsible  for  slating  requirements 
that  lead  to  ideas,  concepts,  and  systems.  They 
play  an  important  role  in  conducting  operational 
tests  and  tactics  development  and  in  supporting 
the  operational  test  and  evaluation  of  majot  new 
systems.  Operational  commands  will  take  the 
systems  into  combat  if  we  go  to  war.  and  they 
will  win  or  lose  depending  upon  their 
effectiveness.  Consequently,  thev  have  a  high 
degree  of  interest  in  all  systems  and  must  alway  s 
play  a  highly  active  role  in  the  acceptance  or 
rejection  process. 
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AFEWES  and  REDCAP:  Indoor  EC  Ranges 


By  lx  CoL  Al  Bryson,  AFOTEC/OL-A1 
INTRODUCTION 

(apt.  Golden  Arm  is  one  of  ihe  Air  Force's 
lop  F-lo  drivers  stationed  at  Briarpatch  AFB. 

Today,  however,  he  is  strapped  in  (he  eoekpit  of 
an  F-  lo  living  over  enemy  territory.  His 
Airborne  Sell  Protection  Jammer  ( ASPJ)  is 
interlaced  with  his  Radar  Homing  and  Warning 
Receiver  (  RHWR).  and  both  are  in  the  green 
Suddenly,  the  Surface-to-Air  Missile  (SAM)  radar 
that  has  been  electronically  painting  his  aircraft 
locks  on.  generating  that  familiar,  yet  hair- 
raising  tone  announcing  danger.  He  checks  the 
alphanumeric  on  the  RHWR  receiver  showing  the 
threat  at  one  o'clock.  The  ASP.I  electronically 
counters  the  threat,  causing  the  SAM  operator 
on  the  ground  to  activate  various  EC'CM  modes 
to  reacquire  the  F-lb  target.  Suddenly  a  missile 
launch  indication  is  received  the  F-lb  cockpit. 

C  ioldcn  frantically  searches  at  one  o'clock  for  a 
visual  on  the  missile  as  he  begins  maneuvers. 

The  SAM  operator  skillfully  maintains  track  on 
the  jinking  target  to  keep  the  missile  on  its 
deadly  intercept  course.  Suddenly.  Golden  has  a 
tally  on  the  streaking  missile  and  in  desperation 
attempts  a  last  maneuver  to  avoid  impact... 

No.  this  is  not  a  paragraph  out  of  Red  Storm 
Rising.  This  is  a  possible  scenario  from  the  Air 
Force  Electronic  Warlare  Evaluation  Simulator 
(AFEWES),  located  at  Air  Force  Plant  #4 
(General  Dynamics).  Ft.  Worth.  Texas.  I  could 
have  just  as  easily  described  an  ingress  of  100 
Blue  aircraft  into  Eastern  Europe,  and  (he 
reaction  that  the  Integrated  Air  Defense  System 
( IADS)  would  take  to  counter  the  invading  force. 
That  would  represent  a  typical  scenario  from  the 
Red  Capabilities  (REDCAP)  test  facility  located 
al  the  CALSPAN  Corp..  Buffalo.  N.Y. 

GENF:RAE  HYBRID  STRENGTHS 

AFEWES  and  REDCAP  are  owned  and  managed 
by  ASD  RWW.  ASD  and  AFOTEC  maintain  on¬ 
site  operating  locations  to  provide  assistance  lo 
both  developmental  and  operational  testers.  The 
operation  and  maintenance  of  the  simulators  are 
performed  bv  contractor  personnel.  Both 
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facilities  use  hybrid,  electronic  combat 
simulations.  Whafs  a  hybrid?  In  simple  terms 
it  means  that  part  of  the  simulated  scenario  is 
accomplished  with  real  equipment,  in  real  lime, 
with  a  man-in-the-loop.  The  remainder  of  the 
simulation  (RCS.  RF  in  free  space,  etc.)  is 
modeled  in  real  time.  The  resulting  hybrid 
simulation  has  both  advantages  and  disadvantages 
when  compared  (tv  range  testing,  such  as  the  lest 
range  at  Eglin  AFB.  FL.  Some  of  its  advantages 
are: 

Security  -  Since  emissions  are  contained  with 
a  secured  and  controlled  environment,  there  is 
no  opportunity  for  compromise  of  sensitive 
information.  Therefore.  SAR  (and  SCI  at  the 
AFEWES)  data  can  be.  and  routinely  is.  used. 

Repeatability  -  Scripted  encounters  can  be 
repeated  numerous  times  under  the  same  set  ol 
test  conditions.  This  allows  the  tester  to 
systematically  alter  test  variables  under  a 
controlled  scheme  to  do  sensitivity  testing  or 
technique  optimization. 

Scoring  lnsir m mentation  -  Each  facility  is  easy 
lo  instrument  due  to  the  very  nature  of  hybrid 
testing.  A  variety  of  MOEs  can  be  selected  to 
meet  the  test  criteria.  Real-time  measurements, 
at  many  points  ol  (he  signal  path  or  C3  net.  can 
be  made  and  recorded  on  strip  charts  or  stored 
in  computer  memory  for  display  in  almost  any 
format  the  tester  desires.  Each  scenario  run 
produces  a  vast  amount  of  data,  and  many  runs 
can  be  made  each  hour  of  testing. 

Pre-Fabrication  Testing  -  Ideas  for  proposed 
systems  can  be  evaluated  against  specific  threat 
systems  or  scenarios  to  assess  potential  prior  to 
large  hardware  development  investments.  Also, 
nonflightworthy  equipment,  such  as  brass  boards, 
can  be  evaluated. 

Signal  Density  -  Both  facilities  can  generate 
large  numbers  of  RF  signals  prov  iding  a  realistic 
background  for  equipment  under  test.  Most  EC 
systems  have  to  process  and  make  decisions  on 
hundreds  of  thousands  of  pulses  each  second. 


Hybrid  simulator''  arc  the  only  places  that 
realistic  pulse  density  tests  can  he  conducted. 

()l  course  there  are  disadvantages  ol  hv  hr  id 
testing.  The  main  ones  are  that  the  equipment 
mulct  test  is  not  in  advcisc  (light  conditions  and 
I  he  lineal  simulation  is  not  transmitting  into 
free  space.  These  are  areas  where  o|>cn  air 
ranees  are  scry  capable . 

COMPLEMENTING  OPEN  AIR  RANGES 

Now  llu  naluial  thine  to  do  is  to  hold  I  he 
In  hr  id  lab  and  the  lest  ranee  up  side  -by-side 
and  do  a  comparison  to  determine  which  is  the 
single  best  place  to  test  Let  me  suggest  that 
\  e  not  compare  their  individual  capabilities,  hut 
that  we  look  at  how  they  compliment  each  other 
Although  the  relative  ratines  on  the  chart  below 
can  be  debated,  it  shows  that  where  one  type  ol 
lest  eapabililv  is  weak,  the  other  has  strength. 
The  relore,  the  logical  way  to  lest  EC  equipment 
is  not  |o  cleeide  whether  to  lake  it  to  a  hybrid 
or  a  ranee,  but  to  determine  how  to  take 
advantage  ol  the  strengths  of  each  by  planning 
an  inteerated  test. 
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AFEWES  VS.  REDCAP 

I  have  referred  to  both  the  A FTW  EiS  and 
RFIX  AP  as  In  hr  ids.  however,  they  are  not 
redundant  capability.  As  in  the  examples  at  the 
start  ol  this  article.  AFEWES'  primary  mission  is 
to  lest  real  F(  M  equipment  on  board  designated 
blue  aircrall  aeainst  a  point  delense  such  as  a 
SAM.  AAA.  or  Airborne  Interceptor.  RLD<  A  P's 
mission  is  to  look  at  a  broader  picture.  How 
does  an  aim. ill.  or  multiple  aircrall.  penetrate 
an  IADS  ’  W  here  are  the  reportine  delays,  and 
what  is  the  value  of  jamming  EW  ( i(  I  assets  or 
cullini!  a  communication  link.’  Typically,  results 
from  A I  I  W  FS  (or  ranee)  testing  can  Iced 
directlv  into  tile  RIJN  AP  test. 


AFEWES  CAPABILITIES 

AEEW'ES  has  high  fidelity,  closed  loop 
simulations  ol  many  of  the  wartime  threats  to 
aircrall  of  both  L 1 S  and  friendly  foreign  lorccs. 

That  is  why  many  US  Army.  Navv.  and  foreign  1 
testers  join  the  Air  Force  in  doing  RD.  DT.  and 
OT  A  E  at  the  AFEW  ES.  Each  of  the  closed-loop 
simulators  typically  has  integrated  transmitter, 
receiver,  signal  processor,  display  and  weapon 
elements  in  hardware.  These  simulators  arc 
designed  to  do  el Icctivcncss  evaluations  and  have 
complete  end-game  seining  eapabililv.  That  is. 
they  can  give  missile  artillery  missed  distance  in 
real-time  and  Pk  via  oil  line  modeling. 

The  AFEWES  simulators  have  a  great  deal  ol 
flexibility,  allowing  a  tester  to  ask  "what  if 
questions.  Such  questions  as  "what  if  the  Red 
system  was  mounted  on  a  .VI  meter  lower  rather 
than  at  ground  level?  "What  il  the  channels  on 
the  threat  receiver  had  a  three  db  imbalance  due 
to  needed  maintenance?"  "W'lial  il  that  was  a 
six  db  or  two  db  imbalance?"  "W'lial  if  1  reduced 
the  RCS  on  selected  Blue  aircraft  .’  "  This  type 
ol  questioning  can  save  time  and  money  lor  the 
tester  who  knows  the  right  question  to  ask  and 
how  to  interpret  the  answer. 

Most  ol  these  simulations  have  been  through 
an  FTD  simulator  validation  (SIMVAL)  and. 
undergo  periodic  upgrade,  to  align  them  with  tile 
latest  intelligence  estimates.  As  with  all 
simulators,  there  are  differences  between  the 
current  simulators  and  the  latest  intelligence.  A 
simulator  validation  does  not  validate  that  the 
simulator  is  exactly  the  same  as  the  real  threat. 

A  SIMVAL  report  does,  however,  tell  a  tester 
the  differences  and  explain  the  potential  impact 
on  lestinir. 

AFEWES  oilers  several  other  significant  test 
capabilities  to  support  the  above  simulators: 

Multiple  Emitter  ( icncrator  (MEG)  -  Capable  ol 
generating  up  ti  2(15  emitters  at  175  sites.  It 
can  generate  any  signal  in  the  .5  to  IS  (ill/ 
range.  The  ME(i  is  the  heart  ol  the  AFEWES 
eapabililv  to  test  the  modern  power  managed  F( 
systems. 

Jammer  Technique  Simulator  -  JETS  can 
simulate  most  ty  pcs  ol  EC  M  in  the  2-  IS  ( ifl/ 
ranee. 

IR  Optical  Tracking  -  RF  tests  comprise  the 
majority  of  the  AFEWES  testing,  however,  there 
are  capabilities  in  other  frequencies. 
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C2  C3  Simulation  -  Simulates  brigade, 
battalion,  and  regimental  headquarter  control 
over  some  ASM  battery  simulations.  Also,  can 
evaluate  RF  jamming  effects  on  voice  and  data 

links. 

Flight  Simulator  Lab  -  This  ultra-high  visual 
resolution  domed  flight  simulator  is  the  properly 
of  (iencral  Dynamics.  However,  it  is  integrated 
with  the  AFEWES  to  do  interactive  tactics 
development  and  Blue  maneuver  evaluation. 

REDCAP  CAPABILITIES 

REDCAP,  like  the  AFEWES,  employs  man-in- 
the-loop.  real-time  simulation.  A  simplistic 
comparison  of  AFEWES  and  REDCAP  would 
show  that  AFEWES  C3  capability  supports  its 
threat  radar  simulators,  and  REDCAP's  EW/CiCI 
radar  simulators  support  its  C3.  The  locus  of 
REDCAP  testing  is  to  evaluate  the  penetration  of 
area  defenses.  This  can  involve  employment  of 
tactics.  ECM.  force  mixes,  and/or  C3CM  to 
determine  the  best  mix  of  EC  elements  within  a 
total  force  structure.  REDCAP  is  generally  very 
useful  whenever  a  penetration  concept  affects 
more  than  one  element  of  an  air  defense  system, 
where  there  are  a  number  of  independently 
acting  penetrators.  or  where  there  are  a 
multitude  of  simultaneous,  coordinated  threat 
signals  present. 

Tiie  I’Mi  addition  of  a  SUAWACS  simulation 
extends  the  EW  (iCI  capability  of  REDCAP  to 
the  air.  This  added  dimension  allows  the 
REDCAP  testers  to  explore  questions  that  can 
not  be  addressed  a!  any  other  test  facility  or 
range  in  the  U.S. 

The  ellecls  and  actions  in  a  large  scale 
netted,  multi-radar,  multi-penetralor  scenario  are 
extremely  complex.  Therelorc.  a  complete  time 
history  of  significant  parameters  and  events  is 
recorded  lor  later  studv  and  lest  analysis.  The 


table  below  is  a  partial  list  of  the  ty  pical 
measures  of  effectiveness  that  are  used  in 
REDCAP  reports: 

EW  Radar  and  Reporting  MOEs. 

-Probability  of  initial  target  detection. 
-Tracking  accuracy. 

-Repotting  rate,  frequency,  delays. 

-False  target  reports,  number,  duration. 

Processing  Center  MOEs: 

-Track  accuracy  versus  number  of  tracks. 
-Ability  to  correlate  redundant  tracks. 

-Delay  from  first  report  to  established  track, 
-Tracking  accuracy  through  triangulation, 
-Ability  of  the  net  to  correlate  sparse  data  to 
direct  weapons. 

Weapons  Direction  MOEs: 

-Assignment  delays. 

-Probability  of  arrival, 

-Probability  of  detection, 

-Probability  of  conversion. 

-Engagement  type,  range,  aspect  distribution. 
-Vectoring  error. 

FUTURE  UPGRADES 

The  basic  capabilities  at  both  AFEWES  and 
REDCAP  need  major  upgrades  to  represent 
potential  weapon  and  C3  systems  that  our  forces 
and  aircraft  will  have  to  counter  in  the  l^'XIs. 

Air  Staff.  SAF/AOL).  has  directed  that  these 
upgrades  be  started  immediately  for  delivery  in 
the  FYVtl-0.3  timeframe.  For  more  information 
on  the  current  and/or  future  capabilities  of  the 
AFEWES  and  REDCAP  to  support  your  test, 
contact  AFOTEC/OL-AI.  Lt  Col  Brvson.  A  V  S3S 
5S54. 
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Lt.  Col.  Allen  L.  Bryson  is  Director  of  the  AFOTEC  Operating  Location  Al. 

AF  Plant  #4.  Ft.  Worth.  Texas.  His  responsibilities  include  planning  lor 
simulator  upgrades  and  facilitating  test  conduct  in  support  of  operational 
testing  at  both  the  AFEWES  and  REDCAP  facilities.  His  previous 
assignments  have  included  duly  in  the  B-52D  ( 1(>‘>  combat  missions  in  SEA). 

RC-145V  M  ( 1 17  operational  missions).  Flight  Commander  at  the  AF 
Electronic  Warfare  Training  School.  Senior  Military  ELINT  Analyst  lor  D1A 
Warnings  and  Indications,  and  ASD  Program  Manager  for  the  AFEWES  and 
REDCAP  facilities.  Lieutenant  Colonel  Bryson  holds  a  bachelor's  degree  in 
Chemistry  from  Southern  Na/arene  University  and  a  master's  ol  System  Management  from  the 
University  of  Southern  California.  He  has  been  awarded  two  Distinguished  Flying  Crosses,  the  Defense 
Meritorious  Service  Medal,  two  Meritorious  service  Medals,  and  thirteen  Air  Medals. 
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Relational  Database  Design 


By  Capl.  Jeffrey  A.  Simmers, 

Del  2  AFOTFC/ASPJ,  Eglin  AFB,  FL 

INTRODUCTION 

Databases  are  more  than  just  organized  data. 

A  functional  database  contains  both  the  plan  for 
storing  the  data,  typically  called  the  database, 
and  the  application  software  that  utilizes  (he 
data.  This  paper  will  call  the  application 
software  database  management  software, 
regardless  of  whether  it's  user  written  or  a 
commercial  program.  The  primary  topic  of  this 
paper  is  the  design  of  relational  databases,  which 
are  databases  whose  data  structures  follow  the 
relational'  model.  Although  this  paper  does  not 
cover  database  management  software  in  any 
detail,  more  information  can  be  found  in 
Reference  I. 

RELATIONAL  DATABASES 

Database  architectures  are  usually  classified 
into  three  major  categories:  relational, 
hierarchical  (tree),  and  plex  (network).  Another 
category,  called  the  inverted  list,  is  actually  an 
indexing  scheme  that  can  be  used  within  any  of 
the  three  and  is  not  generally  considered  a 
separate  architecture.  This  article  addresses 
database  development  within  the  relational 
architecture,  stressing  the  importance  of 
normalization.  ( Ref  I ) 

One  of  the  primary  characteristics  of  a 
relational  database  is  the  storage  of  data  in  one 
I  or  more  flat  files.  A  flat  file  stores  data  in  two 
I  dimensional  tables  (rows  and  columns).  The 
tables  are  structured  so  that  (he  information 
about  the  relationships  between  data  items  is 
preserved,  and  the  term  'relation'  can  be  applied 
to  the  tables.  These  relations,  or  tables  of  data, 
have  caused  the  database  management  programs 
that  use  such  structures  to  be  called  relational 
database  managers  (rDBMS). 

Another  equally  important  concept  behind 
relational  databases  involves  the  concept  of  data 
normalization.  Normalization  can  be  summarized 
as  the  building  of  the  tables  such  that  not  only 
is  all  of  the  information  regarding  the 
relationships  preserved,  but  that  data  redundancy 
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is  kept  to  a  minimum,  and  the  table  rows  and 
columns  can  be  v  iewed  in  any  sequence  without 
affecting  the  functions  that  use  the  tables. 

Often  literature  explains  normalization  by 
constraining  primary  keys  so  that  each  different 
key  value  must  uniquely  identify  one  row  of  the 
table  and  each  key  cannot  contain  any  redundant 
information.  Implementing  primary  keys  in  this 
manner  is  the  usual  approach  to  achieving 
normalization,  and  will  be  demonstrated  (along 
with  the  definition  of  what  a  primary  key  is)  in 
the  examples  in  the  following  section. 

DATA  RELATIONS, 
NORMALIZATION  AND  KEYS 

Although  this  article  targets  relational 
database  management,  many  of  (he  concepts  and 
database  building  examples  can  be  applied  to  tree 
and  plex  database  architectures,  because  their 
structures  can  be  represented  as  flat  files.  (Ref 
D 

Consider  the  test  team  that  wishes  to  build  a 
database  to  store  and  manage  the  data 
represented  in  Table  I.  The  first  question  the 
team  analyst  asks  is. 


will  the  team  have  the  CRUD  (create,  retrieve, 
update,  and  delete)  requirement  for  any  or  all  ol 
the  data?  If  so.  then  building  a  normalized  flat 
file  database  and  using  a  relational  database 
manager  (such  as  DBASE  III.  CONDOR,  or 
R:BASE  V)  should  certainly  be  considered. 
Assuming  the  team  decides  to  follow  such  a  plan, 
let's  step  through  the  basic  procedure  to  build 
such  a  database. 

Table  I  shows  a  flat  arrangement  of  the  test 
team's  effectiveness  data  into  rows  and  columns. 


Usually  the  rows  are  called  records,  and  the 
columns  are  called  fields.  The  database 
management  software  accesses  the  rows  using 
record  numbers,  and  the  fields  through  field 
identifiers  or  titles.  Table  1  shows  the  field 
lilies,  such  as  Mission  Number,  but  docs  not 
show  the  record  numbers,  which  are  just 
sequential  numbers  given  to  each  row  starting 
with  the  first.  Immediately  we  see  that  the 
table  (or  relation)  contains  two  major  types  of 
data.  The  Mission  Number  and  Flown  Date  fields 
are  administrative,  while  the  System  and  ID 
fields  are  effectiveness.  Obviously,  the 
combination  of  administrative  and  effectiveness 
data  is  redundant  since  the  Mission  Number  and 
Flown  Date  are  needlessly  repeated.  This 
relation  is  not  considered  "normalized"  because 
of  the  redundancy. 

The  test  team  can  eliminate  this  redundancy 
by  separating  the  relation  of  Table  I  into  two 
relations,  shown  in  Tables  2  and  3.  Table  2 
contains  the  administrative  relation,  while  Table 
3  holds  the  'Tfectiveness  relation.  Note  that 
Table  2  still  allows  the  test  team  to  track  the 
date  that  each  mission  was  flown  just  as  was 
originally  done  with  the  Flown  Date  field  in 
Table  I." 


Using  Table  2  we  can  now  introduce  the 
concept  of  a  primary  key.  A  primary  key  is  one 
or  more  fields  used  to  uniquely  identify  each 
record  of  the  relation.  The  Mission  Number  is  a 


logical  primary  key.  since  there  is  only  one- 
record  for  each  Mission  Number. 

Choosing  a  primary  key  for  Table  3  isn't  so 
simple,  since  the  Mission  Number  alone  isn't 
enough  to  uniquely  identify  a  record.  However, 
by  combining  Mission  Number  and  System  we  can 
create  a  primary  key  that  does  meet  the  uniquely 
identify  requirement.  Many  database  managers 
allow  you  to  combine  fields  using  string 
concatenation,  and  further  permit  you  to  convert 
a  non-string  field  into  a  string  field  for 
concatenation.  Combining  Mission  Number  ar.d 
System  to  create  a  primary  kev  causes  us  to 
consider  another  important  aspect  of  primary 
keys,  that  they  should  not  contain  redundant 
information.  An  example  ol  redundancy  within  a 
key  would  be  if  vve  used  Mission  Number  •+ 

Flown  Date  +  System  for  a  primary  key  in  Table 
I  (the  "  +  "  indicates  concatenation).  The  Flown 
date  would  be  redundant,  since  each  mission 
number  only  has  one  possible  Flown  date.  But 
Mission  number  +  System  in  Table  3  docs  not 
contain  redundant  information,  and  so  meets  the 
requirements  for  a  primary  key. 

By  separating  Table  1  into  two  relations,  we 
have  not  only  organized  the  data  more  logically, 
but  have  reduced  our  computer  storage 
requirements  (which  may  not  always  be  the 
case).  As  previously  discussed,  vve  have  also 
normalized  the  data  relations.  There  arc  various 
degrees  of  normalization,  but  this  article  will 
just  address  the  minimum  degree  required  for 
most  dalti  base  work.  Further  information  on 
normalization  can  be  found  in  References  1  and 
2.  Normalization  is  the  organization  of  the 
relations  so  that  each  record  can  be  uniquely 
identified  by  the  primary  key  (no  repeating 
records),  and  that  every  non-key  field  is  fully 
functionally  dependent  upon  the  primary  key  (i.e. 
you  can  logically  search  lor  data  in  any  field  ol 
the  record  by  using  the  primary  key).  Both 
Tables  2  and  3  meet  these  two  normalization 
requirements.  Neither  table  repeats  any  records, 
and  in  both  tables  vve  could  track  down  the 
information  contained  in  the  non-key  fields  by 
logical  use  of  the  primary  key.  For  example,  il 
vve  wanted  to  know  what  the  ID  values  were 
against  the  SADS  II  for  all  missions,  vve  «'uld 
easily  obtain  this  information  using  just  the 
primary  key. 

Another  type  of  key.  called  the  secondary  key  . 
is  sometimes  used  to  allow  the  user  to  extract 
certain  types  of  information  from  a  relation,  or 
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to  logically  lie  two  different  relations  together. 

In  Table  3  a  candidate  secondary  key  would  he 
the  Mission  Number,  as  each  Mission  Number  in 
this  table  could  be  equated  with  a  Mission 
number  in  Table  2.  Usually  the  secondary  key 
does  not  meet  all  of  the  requirements  for  a 
primary  key.  Not  only  does  Mission  Number 
allow  us  to  tie  the  two  tables  together 
(effectively  creating  a  larger  'logical'  table),  but 
it  also  lets  us  extract  information  from  Table  3 
for  each  mission.  A  secondary  key  used  to  tup 
into  another  relation  mav  also  be  called  a 
foreign  key. 

PERFORMANCE 

After  the  user  initially  structures  the  database- 
based  upon  data  relations,  normalization,  and 
primary/secondary  keys,  this  structure  must  be 
revisited  based  upon  several  other  important 
considerations.  The  first  of  these  is  database 
performance,  usually  in  terms  of  optimizing  the 
speed  of  as  many  of  the  planned  database  tasks 
as  possible.  Database  tasks  are  basically  simple 
or  complex  combinations  of  the  CRUD  operations. 
A  simple  task  would  be  the  addition  of  new 
records  into  the  database,  while  a  more  complex 
task  would  be  producing  averages  of  multiple 
fields  w  ithin  multiple  records.  General  guidelines 
for  improving  database  performance  are: 

-With  either  real  or  created  data,  build  the 
initial  data  base  structure. 

-Time  as  many  of  the  planned  tasks  as  feasible 
using  (his  initial  database. 

-Extrapolate  (he  timing  data  lo  account  for 
(he  projected  si/e  of  the  actual  database. 

-Decide  which  tasks  should  be  oplimi/ed. 

-Redesign  the  database  structure  to  increase 
the  speed  of  the  tasks  marked  for 
optimization. 

These  five  steps  can  be  repealed  over  and  over 
until  the  user  is  satisfied  that  no  further 
progress  can  be  made.  i.t-..  this  is  a  recursive' 
process. 

As  an  example,  consider  the  relation  in  Table 
4.  The  primary  task  performed  by  the  lest  team 
on  the  database  will  be  to  calculate  the  total 
amount  of  money  spent  on  all  items.  Using  the 
structure  of  Table  4,  the  team  will  have  their 
database  management  software  multiply  the  cost 
times  the  quantity  for  each  item,  and  then  add 
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up  all  the  subtotals.  Typically,  the  team  enters 
one  new 


TABLE  4 

Cost  Data 

Item 

Cost 

Quantity 

1A2 

5.IX) 

100 

2A1 

7.00 

100 

3A6 

8.00 

100 

item  per  day,  and  calculates  the  total  once  at 
the  end  of  each  month.  The  result  of  their 
timing  runs  on  a  small  lest  data  base  yields  the 
following  data: 


TASK 

TIME 

Enter  one  item 

1  minute 

Total  ten  items 

10  minutes 

Unfortunately,  the  team  projects  that  by  the 
end  of  their  test  they  will  have  at  least  1000 
items  in  their  database.  Extrapolating  the  time 
to  total  the  cost  for  ten  items  to  that  for  1000 
items  yields  10  hours  and  40  minutes!  Obviously 
this  task  is  a  candidate  for  performance 
optimization  through  database  restructure.  Since 
computing  the  total  cost  involves  the  calculation 
of  cost  per  item  limes  number  of  items  for  each 
record,  the  team  decides  to  create  a  new  field 
for  each  record  lo  store  this  quantity.  They 
modify  their  database  management  software  to 
automatically  calculate  this  value  and  enter  it 
into  the  database  once  the  user  has  keyed  in  the 
item  and  its  price.  Table  5  shows  the  new 
database  with  this  new  field  called  Subtotal, 
which  is  a  derived  field,  because  it  is  derived 
from  other  fields  instead  of  keved  in  directlv  In 


the  user.  New  timing  runs  yield  the  following 
results: 


Now  the  projected  time  to  calculate  the  end  of 
month  cost  totals  for  a  1000  item  database  is  cut 
in  half  to  eight  hours  and  20  minutes. 

Although  this  example  may  seem  almost  too 
simple  to  take  seriously,  in  practice  the 
performance  optimization  steps  can  be  quite 
sticky.  Typically  users  will  build  a  database 
before  they  completely  define  the  tasks  to  be 
performed,  complicating  step  two.  Step  o 
requires  some  analysis  to  decide  upon  the  shape 
ol  the  extrapolations  (i.e.  linear  versus  non 
linear  increase  in  time  versus  numbers  of 
records).  Additionally,  step  four  forces  the  user 
to  prioritize  all  ol  the  planned  tasks,  and 
consider  how  and  when  they  will  all  be 
accomplished. 

SECURITY 

Security  includes  two  major  categories: 
security  against  unauthorized  access,  and  data 
integrity.  Both  categories  generate  two 
questions  that  need  to  be  answered:  What 
should  be  protected:  how  should  it  be  protected? 
As  part  of  the  answer  to  the  second  question 
the  user  should  consider  alternative  methods  of 
protection,  and  the  relative  costs  (time,  money, 
effort )  of  each. 

Consider  again  the  database  from  Table  5. 

Our  test  team  has  optimized  the  database 
structure  for  performance,  but  a  security  problem 
centers  around  the  Cost  field.  Since  these  items 
tire  part  of  a  competitive  lest  lor  possible 
procurement  by  the  Air  Force,  the  team  must 
restrict  the  ability  to  extract  the  cost  of  each 
item  to  just  test  team  members.  One  way  to  do 
this  would  be  to  use  password  protection  in  the 
database  management  software  that  allows 
viewing  of  the  Cost  and  Subtotal  fields,  while 
allowing  free  access  to  the  Item  and  Quantity 
fields. 

Another  aspect  of  security  is  data  integrity. 
Returning  to  the  database  of  Table  5.  our  team 
discovers  that  an  erroneous  entry  for  the  Cost 
field  could  result  in  a  substantial  error  in  the 
monthly  cost  total.  The  lack  of  built  in 
safeguards  for  data  integrity  is  a  major 


shortcoming  of  many  locally  (i.e.  by  a  test  team 
or  office)  designed  databases. 

For  our  test  team,  one  solution  to  these 
security  concerns  is  a  restructuring  of  the  single 
data  base  of  Table  5  into  two  separate  relations. 

One  relation  contains  only  the  Item  and  Quantity 
data,  while  the  other  is  identical  to  the  relation 
of  Table  5  but  with  the  addition  of  two  software- 
security  safeguards.  The  first  safeguard  is 
password  protection  for  any  use  of  the  database, 
while  the  second  is  an  edit  check'  on  the  Cost 
field.  This  edit  check  would  check  every  value 
entered  into  the  Cost  field  against  a  rule,  such 
as: 

Cost  <  =  10.00 

Edit  checks,  like  this  one  against  the  Cost  field, 
come  from  a  prior  knowledge  of  the  values  for  a 
given  field,  such  as  our  team's  knowing  that  no 
single  item  can  cost  more  than  ten  dollars.  This 
edit  check  would  catch  some  errors,  but 
obviously  will  not  catch  an  erroneous  item  cost 
of  less  than  ten  dollars. 

The  splitting  up  of  the  database  from  Table  5 
does  not  have  to  be  physical.  The  team  could 
still  have  one  physical  database,  but  modify  their 
database  management  software  so  that  the  users 
of  the  database  would  think  there  were  actually 
two  databases,  with  the  primary  difference  that 
one  database  would  require  a  password  and  the 
other  wouldn't.  If.  however,  the  team  elected  lo 
keep  two  physically  separate  databases,  then 
another  aspect  of  data  integrity  must  be 
considered.  If  the  smaller  database  (Item  and 
Quantity  fields  only)  can  be  changed  by  a  user, 
then  the  team  could  easily  have  the  situation 
where  the  two  databases  don't  have  the  same 
data  in  their  common  fields.  To  prevent  this  the 
team  can  only  allow  the  smaller  database  to  be 
read  from,  not  written  to.  bv  the  user.  Users 
with  the  correct  password  could  update  the  full 
database  (the  one  with  the  costs),  and  then  the 
database  management  software  would 
automatically  update  the  smaller  database. 

One  final  caution  about  data  integrity.  With 
the  proliferation  of  math  co-processor  chips,  the 
probability  of  the  computer  actually  making  an 
arithmetic  mistake  seems  to  have  risen  somewhat. 
This  is  due  to  the  independence  of  operation 
between  the  main  processor  chip  and  the  math 
co-processor  chip,  allowing  the  main  processor  to 
be  unaware  of  a  problem  with  the  math  chip. 
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One  possible  guard  against  arithmetic  error 
would  he  a  technique  devised  (or  the  first 
electronic  digital  computer  built  in  this  country, 
the  ENIAC.  ENIAC  programmers  routinely  ran 
an  arithmetic  intensive  program  (with  a  known 
solution)  through  their  machine  as  a  check  for 
faulty  operation  ( Ref  3).  ( )ur  lest  team  could 
easily  do  the  same  by  periodically  running  the 
cost  totaling  program  on  a  non-changing  baseline 
data  base  to  check  for  numerical  accuracy. 

GROWTH 

Failure  to  adequately  project  the  growth  of  a 
database  can  lead  to  major  complications  as  the 
database  grows  beyond  the  processing  or  storage 
capabilities  of  the  computer  and/or  the  database 
management  software.  Growth  projections  used 
for  performance  analysis  can  yield  valuable 
information  pertaining  to  the  capability  of  the 
present  hardware  and  software  to  handle  the 
future  database.  An  office  or  test  team 
preparing  a  request  for  computer  equipment 
should  consider  if  the  requested  equipment  w  ill 
be  capable  of  meeting  projected  database 
management  requirements. 

If  database  growth  appears  destined  to  outstrip 
the  local  computer  resources,  then  consider 
alternatives,  such  as  the  use  of  a  time-shared 
mainlrame  computer.  One  alternative  would  be 
storing  raw  data  on  a  mainframe  computer,  then 
periodically  downloading  either  processed  data  or 
subsets  of  the  raw  data  for  management  on  local 
PC's  ( personal  computers).  Integrating  different 
databases  in  different  locations  in  this  manner  is 
called  distributed”  database  processing,  which  is 
a  complex  topic  in  itself  (Ref  1). 

Another  approach  to  compensate  for  problems 
due  to  database  growth  is  manipulating  the 
database  structure.  Breaking  up  one  large 
relation  into  smaller  relations,  as  we  did  with 
Table  I.  will  often  solve  the  problem.  For  a 
database  with  substantial  derived  data,  separating 
the  derived  fields  into  a  separate  relation  may 
help.  Unfortunately,  modifications  made  to 
facilitate  growth  considerations  may  adversely 
impact  previous  changes  made  for  other  reasons. 

SUMMARY 

This  paper  presents  guidelines  for  structuring 
the  data  within  the  relational  database  model  to 
facilitate  the  concepts  of  normalization,  kevs. 
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performance,  security,  and  grow  th.  But  database 
structure  is  just  the  first  of  a  two  part  process. 

The  second  step  is  the  design  and  implementation 
of  the  database  applications  using  database 
management  software.  This  software  may  be  a 
commercial  database  manager,  such  as  dBASE  III. 
or  it  may  be  a  specialized  program  written  by 
the  user.  Although  this  paper  does  not  address 
such  software  applications,  database  structure 
and  application  software  cannot  be  considered 
independently.  Before  designing  a  database 
structure,  the  user  should  consider  the 
applications  and  their  impact  on  the  database- 
design.  Reference  2  contains  more  information 
about  the  integration  of  data  and  application. 
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